face_recognize_facenet error (ValueError: bad marshal data (unknown type code))

[juaone1]

Hi Im a newbie and I was trying the face_recognize examples. the face_recognize (without facenet) is very fast but it is unreliable even if I did have multiple photos of the person. how many photos do you recommend for this model to be accurate?

Since the non facenet model is not so reliable I tried to use the facenet example and I encountered this issue.

2023-02-16 18:00:22.606014: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll';
dlerror: cudart64_110.dll not found
2023-02-16 18:00:22.606447: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
File "c:\Users\John Garcia\PycharmProjects\Face Analyzer\FaceAnalyzer\examples\OpenCV\face_recognize_facenet\face_recognize.py", line 29, in

facenet = tf.keras.models.load_model(str(facenet_path))
File "C:\Users\John Garcia\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\John Garcia\AppData\Local\Programs\Python\Python39\lib\site-packages\keras\utils\generic_utils.py", line 857, in func_load
code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

I hope you can help me fix this issue and point me to the right direction. Thank you for this wonderful project that you've share

[ParisNeo]

Hi. and thanks for using FaceAnalyzer.

The face_recognize without face net is a very simple tool to recognize faces just by the dirtributon of the face landmarks. It is simple but as you said, not that reliable as it looses too much information. I have few ideas to enhance it but I just didn't have enough time to do that and I have many other projects that took over this one. But I will eventually make a more robust version.

It did work for me with around 10 images of each person in different orientations. But it highly depends on the similartities between the subjects you are trying to discriminate. There is a script that allows you to take the pictures and it does put the output in the right format.

As of facenet, it looks like you have some issues with your tensorflow installation.

Do you have an nvidea graphics card compatible with cuda? If so, try to reinstall the cuda drivers.
If you use anaconda for python environments, you can install it directly:

conda install pytorch-cuda=11.7 -c pytorch -c nvidia 

Otherwize try reinstalling tensorflow v 2.0 or higher.

Best regards

[juaone1]

Thank you for responding back to me,
I have yet to try to reinstall the tensorflow, im planning to use the facenet model into raspberry pi 4b. Will it be possible?

If you can share or point me into the right direction on how can I make the non facenet model more robust, It would really help me big time.

Thanks again for this very wonderful project.

[ParisNeo]

Hi, I didn't test this on raspberry pi yet. The non facenet version should work with no problem. I just need to enhance the detection code. The facenet version is more sophesticated and I can't really tell if it would work fine on raspberry pi but on PC it gave really good results.

I have created FaceAnalyzer essentially to play with my daughter so these features were just built for specific personal use. And I am happy that people are using this. It is in MIT licence so you can use it, modify it or do whatever you want with it (of course this has to be legal activity). The face detection example was built essentially in one night, so I didn't put that much effort in it. But if I have time I'll be enhancing it and maybe adapting it to Raspberry pis.

In the meanwhile, If you want to go faster, you can use this library:
https://pypi.org/project/face-recognition/

It sa already been tested with raspberry pi. Here is a video that explains this :
https://www.youtube.com/watch?v=EQZ79QkRutY

I hope this helps.
Best regards

[juaone1]

Hi again, I've already tried that face-recognition library and it works well but the only problem is I can't detect and recognize faces that are in side view. In my application I need to recognize face even if its side view and your face recognition works well in detecting the face but cannot recognize it accurately. Is there a way to make it more accurate and robust? Can you point me in the right direction as to how I can improve your example with the non facenet version.

Thanks

[ParisNeo]

Sorry I didn't check my notifications yesterday.

I have made some clarifications in the facenet version. I have tryed it today and it does work fine. You may want to give it a shot.

As for the simple version, you can fork this repository and do the changes. For example instead of using just the landmarks relative positions, you can convert the image to hsv and store the hue part of each pixel under the landmarks. this will make the computation of the distance between face signatures take the skin color into account.

Best regards

[juaone1]

Hi I tried to clone the repo, I've downloaded the facenet_keras_weights.h5 and run the convert_facenet_to_modern.py afterwards I've tried to run the face_record.py and encountered this issue

line 44, in
import tkinter as tk
ModuleNotFoundError: No module named 'tkinter'

how can I resolve this?

--Edit

I removed the tkinter dialog and replace it with simple input. I have successfully saved a face but when I run the face_recognize facenet version I encounter this error.

line 88, in
print(f'Distance between two is :{np.linalg.norm(known_faces[0]["mean"]-known_faces[1]["mean"])}')
IndexError: list index out of range

[ParisNeo]

This is bizarre as tk is a standard python package that is available on all python distributions!
This code needs at least python 3.8 to be executed. Can you check your python version?

You can check this thread on stack overflow

Before executing face recognize you have to store faces to recognize.
Use the face_record script for this matter to store faces to recognize. You need to press s then move your face in front of the camera to store the images. I also used tk for this to simplify the recovery of the face name.

[juaone1]

My python version is 3.9, is there anyway I can install tkinter in windows?

Thats what I've done and it shows the error Index out of range

--Edit

I have managed to make it work but it is still not accurate. When there are two or more faces registered it still cannot recognize me properly.

[ParisNeo]

OK, let me explain how this code works.

I use a neural network trained using many faces. This neural network named facenet, encodes your face in what we call a latent space. This means that for each picture of your face, the neural network outputs a point inside a multidimensional space. With many photos of you in different postures, we get a could of points in that space. The neural network was trained such that when two faces belong to the same person, the output points are close to each other and when the two faces belong to different persons, the output faces are far from each other.

The example of face analyzer is composed in two parts. First the recorder will build the cloud of points and save the mean and the standard deviation of the point cloud in a pickle file for each person. You need to provide multiple face orientations otherwize the standard deviation will be 0

In inference time, face analyzer is used to detect the faces, and each face is encoded using facenet network then we simply find the nearest mean to the current point.

Here I use absolute distance:
distance = np.abs(known_face["mean"]-embedding).sum()
if distance<nearest_distance:

You can try to modify this with an L2 distance np.sqrt(((known_face["mean"]-embedding)**2).sum())

Here I use just a simple distance between the current point and the mean of the points which means that I don't use the standard deviation. This is possible if the database used for the training is biased (for example having more white people than black, asian or latino people), this can cause this kind of problems. I am not the one who trained the facenet network so I don't know if there are any biases.

You can also use a more complicated mahalanobis distance that takes into consideration the distribution of your cloud point but you would need to reduce the space dimensions as you have 128 dimensions and probably less that 128 images which will probably results into a singular covariance matrix which forbids using mahalanobis distance. In this case, you need to use some linear algebra wizardry to reduce the dimension for example using a PCA projection.

I've built an example using mahalanobis distance. I have also added a new script that takes a folder containing the images to learn from so that you can experiment on people images that without using the recording script.

Finally I have built another tool that uses deepface for facial recognition and it seems to work way better than the facenet version.

You must install deepface if you want to use this:
pip install deepface

This is a really cool library. You can find it here:
https://github.com/serengil/deepface

The example I made is similar to the one with facenet. It just uses cosine distance instead. This distance is a kind of angular distance. It goes from 0 to 2 where 0 means similar, 1 means orthogonal and 2 means completely opposite.

I hope this all helps.

[juaone1]

Hi again I have tried the deepface example, I'm confused as to how to use it. Should I use the face_record or face_db_record. When I tried to use the face_record it saves a pkl file but when I run face_recognize there are no bounding boxes or facial landmarks point in the screen. it only displays something like this in the terminal:
1/1 [==============================] - 1s 534ms/step
1/1 [==============================] - 0s 293ms/step
1/1 [==============================] - 0s 299ms/step
1/1 [==============================] - 0s 292ms/step

and when I try to run the face_db_record, I have already put the photos inside the sub folder following the tree structure that you gave. I encounter this error.

1/1 [==============================] - 1s 527ms/step
'float' object is not subscriptable
Traceback (most recent call last):
File "c:\Users\user\PycharmProjects\FA2\FaceAnalyzer\examples\OpenCV\face_recognize_deep_face\face_db_record.py", line 81, in
embeddings_cloud_inv_cov = np.linalg.inv(np.cov(embeddings_cloud.T))
File "<array_function internals>", line 180, in inv
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\linalg\linalg.py", line 539, in inv
_assert_stacked_2d(a)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\numpy\linalg\linalg.py", line 196, in _assert_stacked_2d
raise LinAlgError('%d-dimensional array given. Array must be '
numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional

I'm really sorry for all the questions

[ParisNeo]

You need to record faces before recogniezing them.
You now have two ways todo this:
1 - use face_recognize.py to record your own file from the webcam. I think that you didn't record enough valid photos. I just made an update to handle this issue. You should press s and then move your head left and right untill the progressbar is filled in your console.
2 - use face_db_record. In this case, you can put multiple photos of the persons you want to add to the database inside the subfolder faces_db. You build a subfolder for each person and put a bunch of images of the faces of these people. Makesure there is one face for each photot and the face is not too far from the camera. You can check the images extracted by the algorithm by looking inside the generated subfolder faces_db_preprocessed.

Once this is done, now you can run the recognize script and it should work. It worked very fine for me.

You can also look at the deep_face repository for more examples. This tool does more than just recognition.

I am building a new example that uses gradio to fuse every thing in a simple web based UI for face recognition. The first version is already live, you need to install gradio :
pip install gradio

then run the code and you open your web browser and go to http://127.0.0.1:7860. You'll find an easy to use UI that shows every thing.

For now I only implemented the face recognition, so you need to use the face recording code to build the database. But I'll be adding this t the ui so that you can track and add new faces

[juaone1]

When using the deepface version is it really needed to have a gpu? when I tried recording my face it does not detect it.
0%| | 0/50 [00:08<?, ?it/s]
Traceback (most recent call last):
File "c:\Users\user\PycharmProjects\FA3\FaceAnalyzer\examples\OpenCV\face_recognize_deep_face\face_record.py", line 111, in
embedding = DeepFace.represent(image)[0]["embedding"]
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\deepface\DeepFace.py", line 754, in represent
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\deepface\commons\functions.py", line 178, in preprocess_face
img, region = detect_face(img = img, detector_backend = detector_backend, grayscale = grayscale, enforce_detection = enforce_detection, align = align)
File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\deepface\commons\functions.py", line 124, in detect_face
raise ValueError("Face could not be detected. Please confirm that the picture is a face photo or consider to set enforce_detection param to
False.")
ValueError: Face could not be detected. Please confirm that the picture is a face photo or consider to set enforce_detection param to False.

its the same when I use the face_db_record script, it does not detect a face in all of my photos.

[ParisNeo]

That's wild. At least this should work on GPU. I have a live demo on hugging face that actualy work

[ParisNeo]

Try the gradio version. Maybe you can have better luck with that one?
Instead of going to the opencv examples, you can try the gradio examples.

[ParisNeo]

there is a live demo on hugging face:
https://huggingface.co/spaces/ParisNeo/FaceRecognition

[juaone1]

hi its error 404 on hugging face

[ParisNeo]

sorry, I removed it this morning because one of my friends detected that the images posted by people were visible by other people and since I am working now I can't fix this so I converted the repo to private. I'll bring it back to live maybe this evening when I repare the face sharing issue.

Sorry for that.

The code is still in the examples of faceanalyzer. It is in gradio subfolder..

You can also take a look at the deepface project. You can use it without needing to use faceanalyzer if you want to try some alternatives.