BOM sometimes is located INSIDE text, not at the beginning — if a file has been assembled some time by php from other files using for example include_once(). To remove it, delete area between at least one character before BOM and at least one character after BOM (just in case). Position of BOM can be located in F12 Developer Tools of the Internet Explorer and probably Edge. It is visualised as a black diamond / rhombus.
Visual Studio and WebMatrix can save files with or without signature (at the beginning).
BOM causes errors during validation ( https://validator.w3.org/#validate_by_upload ) or in consoles — </HEAD> can be treated as orphaned element without <HEAD>, when apparently is present !:
Error: Stray end tag head.
<BODY> as second one <BODY>, when only one <BODY> exists and everything is correct:
Error: Start tag body seen but an element of the same type was already
open.
And entire document can be seen lacking DOCTYPE, when BOM or two BOMS occupy first line and DOCTYPE is in second line, with a message similar to this one:
Error: Non-space characters found without seeing a doctype first.
Expected e.g. <!DOCTYPE html>.Error: Element head is missing a required instance of child element
title.Error: Stray doctype.
Error: Stray start tag html.
Error: Stray start tag head.
Error: Attribute name not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Attribute http-equiv not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Attribute name not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Element link is missing required attribute property.
Error: Attribute name not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Attribute name not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Attribute name not allowed on element meta at this point.
Error: Element meta is missing one or more of the following
attributes: itemprop, property.Error: Element title not allowed as child of element body in this
context. (Suppressing further errors from this subtree.)Error: Element style not allowed as child of element body in this
context. (Suppressing further errors from this subtree.)Error: Stray end tag head.
Error: Start tag body seen but an element of the same type was already
open.Fatal Error: Cannot recover after last error. Any further errors will
be ignored.
( https://validator.w3.org/#validate_by_uri )
And stream of messages in IE F12 Developer Tools console:
HTML1527: DOCTYPE expected. Consider adding a valid HTML5 doctype: «<!DOCTYPE html>».
HTML1502: Unexpected DOCTYPE. Only one DOCTYPE is allowed and it must occur before any elements.
HTML1513: Extra «<html>» tag found. Only one «<html>» tag should exist per document.
HTML1503: Unexpected start tag. HTML1512: Unmatched end tag.
Everything caused by one BOM at the beginning. And Debugger shows one black rhombus in the first line.
Files saved with signature, but not assembled by php don’t cause such errors and black diamonds are not vissible in IE debugger. So perhaps php transforms BOM somehow. It seems that main php file must be saved with signature to see this.
Those strange characters occur at the beginning and/or on the borders of files merged with include_once() and are not visible when files are saved before without signature. This is why it points at BOM involvement.
I have noticed this everything day before yesterday when started converting my website to HTML5 and validating.
BOM can also create a small indent at the beginning of line. Two files containing identical text but one with indent.
With Python, it is a really easy to retrieve data from 3rd party API services, so I made a script for this purpose. The script worked without any issue for many different API URLs, but recently, when I wanted to load the server content response from a specific API URL into json.loads method, it threw an «Unexpected UTF-8 BOM» error. In this article, we will examine what the error means and various ways to solve it.
To retrieve the data from 3rd party API service, I use this code in my Python script:
import requests
import json
url="API_ENDPOINT_URL"
r = requests.get(url)
data = json.loads(r.text)
#....do something with the data...
The above code uses requests library to read the data from URL and then it uses json.loads method to deserialize a server’s string response containing JSON data into an object.
Until this particular case, the above code worked just fine, but now I was getting the following error:
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
The error was caused by the json.loads(r.text), so I examined the value of r.text, which had this:
ufeffn{retreived data from the api call}
The content from server’s response contained the data from the API, but it also had that strange ufeff Unicode character at the beginning. It turns out, the Unicode character with value u+feff (or xefxbbxbf in binary) is a byte order mark (BOM) character.
What is BOM
According to Wikipedia, the BOM is an optional value at the beginning of a text stream and the presence can mean different things. With UTF-8 text streams, for example, it can be used to signal that the text is encoded in UTF-8 format, while with UTF-16 & UTF-32, the presence of BOM signals the byte order of a stream.
In my case, the data was in UTF-8 and has already been received, so having that BOM character in r.text seemed unnecessary and since it was causing the json.loads method to throw the JSONDecodeError, I wanted to get rid of it.
The hint on how to solve this problem can be found in the Python error itself. It mentions «decode using utf-8-sig«, so let’s examine this next.
What is utf-8-sig?
The utf-8-sig is a Python variant of UTF-8, in which, when used in encoding, the BOM value will be written before anything else, while when used during decoding, it will skip the UTF-8 BOM character if it exists and this is exactly what I needed.
So the solution is simple. We just need to decode the data using utf-8-sig encoding, which will get rid of the BOM value. There are several ways to accomplish that.
Solution 1 — using codecs module
First, I tried to use a codecs module which is a part of a Python standard library. It contains encoders and decoders, mostly for converting text. We can use the codecs.decode() method to decode the data using utf-8-sig encoding. Something like this:
import codecs
decoded_data=codecs.decode(r.text, 'utf-8-sig')
Unfortunately, the codecs.decode method didn’t accept strings, as it threw the following error:
TypeError: decoding with ‘utf-8-sig’ codec failed (TypeError: a bytes-like object is required, not ‘str’)
Next, I tried to convert the string into a bytes object. This can be done using encode() method available for strings. If no specific encoding argument is provided, it will use the default encoding which is UTF-8 (at least on Windows):
decoded_data=codecs.decode(r.text.encode(), 'utf-8-sig')
data = json.loads(decoded_data)
The decoded_data variable finally contained data without the BOM byte order mark Unicode character and I was finally able to use it on json.loads method.
So, this worked, but I didn’t like I was using an extra module just to get rid of one Unicode BOM character.
Solution 2 — without using the codecs module
It turns out, there is a way to encode/decode strings without the need of importing codecs module. We can simply use decode() method on the return value of string.encode() method, so we can just do this:
decoded_data=r.text.encode().decode('utf-8-sig')
data = json.loads(decoded_data)
Let’s try to simplify this further.
Solution 3 — using requests.response content property
So far, the code in this article used r.text that contains Request’s content response in a string. We can skip the encoding part all together by simply using the r.content instead as this property already contains the server content response in bytes. We then just simply use decode() method on r.content:
decoded_d=r.content.decode('utf-8-sig')
data = json.loads(decoded_data)
Solution 4 — using requests.response encoding property
We can skip the part of calling encode() and decode() methods as shown in previous examples all together and instead use the encoding property of a requests.response object. We just need to make sure we set the value before the call to r.text as shown below:
r.encoding='utf-8-sig'
data = json.loads(r.text)
Conclusion
If the json.loads() method throws an Unexpected UTF-8 BOM error, it is due to a BOM value being present in the stream or a file. In this article, we first examined what this BOM is, then we touched a bit about utf-8-sig encoding and finally, we examined 4 ways to solve this problem.
Время на прочтение
2 мин
Количество просмотров 2.3K
Столкнулся с проблемой некорректного отображения кириллических шрифтов в браузере, а точнее браузер неправильно определял кодировку. Краткий анализ показал, что данное неудобство проявляется только при включении плагина ZF debug. Кинув взгляд на исходный код страницы увидел, что стили и скрипты свои плагин подключает сразу же после открывающего тега <head>, то есть до метатега с информацией о кодировке страницы, что, видимо, не совсем правильно.
Для исправления ситуации необходимо подправить файл libraryZFDebugControllerPluginDebug.php следующим образом(выделено черным цветом)
protected function _headerOutput() {
$collapsed = isset($_COOKIE[‘ZFDebugCollapsed’]) ? $_COOKIE[‘ZFDebugCollapsed’] : 0;return (‘
<style type=»text/css» media=»screen»>
…
</script> </head>‘);
}
protected function _output($html)
{
…
$response->setBody(preg_replace(‘/(</head>)/i’, ‘$1’ . $this->_headerOutput(), $response->getBody()));
…
}Все, теперь
<meta http-equiv="content-type" content="text/html; charset=utf-8" />будет стоять сразу после открывающего тега<head>.п.с. Проблема так же решается если принудительно указывать BOM в начале файла, но, к примеру, PHP Storm не умеет(на данный момент) сохранять его. Однако же BOM не необходим для браузерных приложений, даже излишен, исходя из документа «Use of BOM is neither required nor recommended for UTF-8»
In Python, You will get an error while retrieving the data from any 3rd party API request. In fact, when response content converts to JSON format using json.loads method, it throws an json.decoder.JSONDecodeError: Unexpected UTF-8 BOM error. In this article we are going to see how to fix json.loads() Unexpected UTF-8 BOM error in Python.
How to Fix json.loads Unexpected UTF-8 BOM error in Python. We have seen solutions to fix Unexpected UTF-8 BOM errors when using json.loads in Python.
The error was occur by the json.loads(r.text), so when text content convert to JSON format we have getting following error:
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
The response content coming from the API, but ufeff Unicode character coming at the beginning. This Unicode character value ufeff (or xefxbbxbf in binary) is a byte order mark (BOM) character.
Python: Fix json.loads Unexpected json.decoder UTF-8 BOM Error
Following are 4 different solutions. Basically all following solutions we have to decode the data using utf-8-sign encoding. This way we can fix the error.
Solution 1 Decode content using utf-8-sig
In this solution, we can use decode() method on the return value of the string.encode() method. This is the most efficient solution to fix this error.
decoded_data = r.text.encode().decode('utf-8-sig')
data = json.loads(decoded_data)Solution 2 Decode response content
This solution is a straightforward method to fix this issue. We have used the decode() method on r.content.
decoded_data = r.content.decode('utf-8-sig')
data = json.loads(decoded_data)Solution 3 Encode requests.response object
In this solution, you can use the encoding property on the response object. This way we can skip the previous examples showing the calling of encode() and decode() methods.
r.encoding = 'utf-8-sig'
data = json.loads(r.text)Solution 4 Use Python codecs module
You can use the Python codecs module. We can use the codecs.decode() method to decode the data using utf-8-sig encoding. The codecs.decode() method accepts a bytes object. Thus, you have to convert the string into a bytes object using encode() method.
decoded_data = codecs.decode(r.text.encode(), 'utf-8-sig')
data = json.loads(decoded_data)Bottom Line
All in all, if the json.loads() method throws an unexpected UTF-8 BOM error. It means BOM values are existing in the response data. In this article we have seen 4 different solutions to rid out this json.loads Unexpected UTF-8 BOM error in Python.
Another most common error: object arrays cannot be loaded when allow_pickle=false. You can check fixexception to find an appropriate solution.
We hope you have found this article helpful. Let us know your questions or feedback if any through the comment section in below. You can subscribe our newsletter and get notified when we publish new articles. Moreover, you can explore here other interesting articles.
If you like our article, please consider buying a coffee for us.
Thanks for your support!
Buy me a coffee!
-
Помощь -
Хостинг -
Сайт не работает -
Что такое BOM символы и как с ними бороться
При создании и редактировании файлов сайта с помощью стандартных программ, редакторы могут автоматически присвоить вашему файлу кодировку UTF-8 с BOM меткой.
BOM (Byte Order Mark) — символ вида U+FEFF, увидеть который можно в самом начале текста.
К чему приводит наличие символа BOM
- в файлах с расширением php часто высвечивается ошибка:
Warning: Cannot modify header information — headers already sent by (output started at …
- в файлах с расширением html сбиваются настройки дизайна, сдвигаются блоки, могут появляться нечитаемые наборы символов.
Чтобы исправить это, нужно пересохранить файл с кодировкой UTF-8 без BOM.
Первый способ
-
1.
Откройте файл с помощью редактора Notepad++.
-
2.
Нажмите Кодировки — Кодировать в UTF-8 (без BOM):
Второй способ
-
1.
Подключитесь к серверу по SSH: Как подключиться по SSH?
-
2.
Выполните команду для проверки всех файлов на наличие в них символов BOM:
find -type f|while read file;do [ "`head -c3 -- "$file"`" == $'xefxbbxbf' ] && echo "found BOM in: $file";doneЕсли хотите проверить только определенную директорию, то перейдите в нужный каталог.
-
3.
Если такие файлы есть, запустите следующую команду для удаления символов BOM:
find . -type f -exec sed 's/^xEFxBBxBF//' -i.bak {} ; -exec rm {}.bak ;
Спасибо за оценку!
Как мы можем улучшить статью?
Нужна помощь?
Напишите в службу поддержки!