Replies: 16 comments 70 replies
-
|
Hello, Why not. However, I need to get my hand on the hardware (which should not take too much time). Also, don't forget that we are messing with I don't exactly know how much time it can take, so… we'll see. |
Beta Was this translation helpful? Give feedback.
-
|
I made some progress yesterday:
My changes are being implemented here. |
Beta Was this translation helpful? Give feedback.
-
|
I have a mission abroad next week, so no progress will be made on my side. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Vincent, thanks for your efforts, have you made any further progress? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @chrismelikian, Sorry, not much progress on my side at the moment. I'll let you know if I get back on it. |
Beta Was this translation helpful? Give feedback.
-
|
@Vincent-Stragier are you still making any progress on this? I found the project and would like to try it with the Pi camera 3. Just ordered the hardware, so I should have it sometime in the next week. |
Beta Was this translation helpful? Give feedback.
-
|
I would be glad to join the effort for this and for the Pi Camera v3 (gotta buy one or two) I am an old hand at development, but kinda new to Python applications. I did my work in C and C++ Let me know, please. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @linuxdan, The challenge is to get the rotating frame buffer, the picture capture and the live view working simultaneously. Maybe the example of the lib have been updated, but I've not looked at the code for a long time. |
Beta Was this translation helpful? Give feedback.
-
|
It might be worth arranging to talk so that you might give me an overview of the application’s current state. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, but first, I'd like to get my hands on back a running system (I'll manage to free some time next WE). |
Beta Was this translation helpful? Give feedback.
-
|
Since starting this thread, picamera2 has evolved a bit. They now have an example which could be helpful to us. I'm keen to also explore if we can do similar motion detection without OpenCV. I'll play around and report back. |
Beta Was this translation helpful? Give feedback.
-
|
The examples need comments that explain what is happening. |
Beta Was this translation helpful? Give feedback.
-
|
Over the last week I've been gradually working my way through the code in this project and updating it to work with Picamera2 (libcamera) and the Pi Camera Module 3. I've reached the point where everything now seems to be working fine, but I do need to run it for a week or so with it capturing before I'm happy there are no memory leaks or other bugs that haven't been picked up in my testing so far. At that point I'll make a pull request. I have added a number of new features, as follows: Option to switch between two resolutions on the web interface (1640x1232 and 1920x1080). MP4Box no longer seems to exist so I've changed the code to use ffmpeg instead, so that will need installing first. My testing on a Pi Zero 2W with the camera module 3 has shown that using framerates above 20 cause the Pi to eventually lock up, so I've set this to 20 in the config.json file. If anyone else wants to test the new code, my fork is located at https://github.com/caracoluk/NaturewatchCameraServer/ and it's commit 36a3103. I'll confirm in a week or so if my testing has been successful. |
Beta Was this translation helpful? Give feedback.
-
|
My testing over the past week hasn't identified any issues so I have just submitted a Pull Request for review. The additional packages required to enable picamera2/libcamera support have been added to the build script. As far as I'm aware, in order to support the Pi Camera Module 3 it is necessary to be running a relatively up to date version of the latest Pi OS kernel, Bookworm. So, a full image rebuild would be the best approach. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for all your work @caracoluk. I'll need to test this on our side, waiting for a pi camera module 3. Will let you know how it goes! |
Beta Was this translation helpful? Give feedback.
-
|
@caracoluk thanks for doing this -- just in time for me to start a summer project with the kids! I have all the hardware ready and can also test it out in the next week or so. @mikevanis I haven't built a custom PiOS image before but the workflow steps seem straight-forward. Is there anything else I would need to know to get it working? |
Beta Was this translation helpful? Give feedback.
-
|
@gschivley if you have problems running the workflow steps for some reason and want to get a test set up in time for the summer break, then let me know as I have a full set of build instructions that I used to build from a fresh install of PiOS Lite BookWorm that I could send over. |
Beta Was this translation helpful? Give feedback.
-
|
Mind sending them over?
…On Tue, Jun 18, 2024 at 4:19 AM caracoluk ***@***.***> wrote:
@gschivley <https://github.com/gschivley> if you have problems running
the workflow steps for some reason and want to get a test set up in time
for the summer break, then let me know as I have a full set of build
instructions that I used to build from a fresh install of PiOS Lite
BookWorm that I could send over.
—
Reply to this email directly, view it on GitHub
<#111 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACPERVC26LKUWB6BT7J4EUDZH7USDAVCNFSM6AAAAAAZXDSUZOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQMBTGMZTA>
.
You are receiving this because you were mentioned.Message ID:
<interactionresearchstudio/NaturewatchCameraServer/repo-discussions/111/comments/9803330
@github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Sure, might be easier if you let me know your email address and I'll send them over that way. |
Beta Was this translation helpful? Give feedback.
-
|
Could you upload them as a GitHub gist or something so we could all have a try please?
Sent from Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: caracoluk ***@***.***>
Sent: Tuesday, June 18, 2024 11:53:14 AM
To: interactionresearchstudio/NaturewatchCameraServer ***@***.***>
Cc: chrismelikian ***@***.***>; Mention ***@***.***>
Subject: Re: [interactionresearchstudio/NaturewatchCameraServer] Looking for help: transitioning to picamera2 (Discussion #111)
Sure, might be easier if you let me know your email address and I'll send them over that way.
—
Reply to this email directly, view it on GitHub<#111 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABBNBDQLW23PFLTZIHVCYATZIAGRVAVCNFSM6AAAAAAZXDSUZOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQMBUHA4DS>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
For those wanting to test the updated code while we await the Pull Request testing, here are the steps required to build from scratch: Install latest version of Raspbian Bookworm (June 2024) onto your Pi Zero W/Zero 2W Add these commands to /boot/firmware/config.txt: Update Pi to latest sw/fw: Install extra packages: Clone the NatureWatch git repository: Install NatureWatch service: Note that this won't set up the MyNatureWatch WiFi, but everything else should be functional. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, and expect that process to take a good few hours to complete if you do decide to do it that way! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the alternative build instructions! I've got it started and will (hopefully) be able to start playing around in the next couple days. FYI, I hit the error below so I did all lower-case for the project name. In case anyone else is trying to follow along. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, I used a different name for my project which was all lower case but when I copied my notes onto here I changed the name to make it more appropriate. One change that I thought wouldn’t matter which clearly does! Just goes to show you have to test everything. I’ve updated my comment above to make it all lower case. I hope the testing goes well. |
Beta Was this translation helpful? Give feedback.
-
|
My pi has been stuck at |
Beta Was this translation helpful? Give feedback.
-
|
That part of the installation took about an hour on my Pi Zero 2W, so it does sound like something isn't right. Was this a fresh install or had you already tried to install react through another means? Just wondering if that has left it in an unknown state after it failed. |
Beta Was this translation helpful? Give feedback.
-
|
I think it does take a while to install opencv-python-headless. I triggered the installation and went to get some lunch. When I came back an hour later it had finished so I don’t know exactly how long it took. I do recall it taking a while to install from previous projects however.On 22 Jun 2024, at 16:39, Greg Schivley ***@***.***> wrote:
How long did it take you to install all the dependencies? opencv-python-headless seems to be an especially slow one.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hi Vincent,
The instructions were really intended as a quick means of testing the code rather than a recommended way of building the configuration. They were the process I followed to get to the point where I could start working on the code however. That’s useful information about the use of a venv and not something I’ve come across before. The use of opencv-python-headless was taken from the original build script and I simply copied that when installing the necessary modules.
As for installing nodejs and compiling, I did perform those tasks on my system but left them off of the instructions as I’d included the compiled project on the GitHub repository. The main repository didn’t have the compiled project in the build directory so I was expecting the compilation to replace those when the install script runs.
I’ve a feeling the /home/pi references might have been left in by accident when I was replacing mp4box with ffmpeg. That shouldn’t be a problem to resolve.
|
Beta Was this translation helpful? Give feedback.
-
|
Don't worry, my message was mainly to point out what may block during the installation (and even, I need to retest my instructions). For nodejs, even in your repo (it's only on your local machine) you won't have the built project as For |
Beta Was this translation helpful? Give feedback.
-
|
Ah, I hadn't spotted that the build directory is excluded when publishing to github. It's just as well you updated the list of instructions to include the process for compiling the project as without that it wouldn't work at all! That's probably the most time consuming part of the installation process, and I think it took 1-2 hours for the nodejs installation and project compilation to complete. I noticed that I also left Flask debugging enabled, and logging set to DEBUG instead of INFO in a couple of places so I do need to clean that up too. I'll need to wait until I've finished some other testing on my Pi Zero 2W as I appear to have an issue with the wireless driver crashing sporadically. It's unrelated to this project and I think (well, hope) I've now found a solution to that issue but want to leave the Pi running a week to make sure it's stable. It would previously only stay up for a day or so. This is my first time uploading code to Github, so I was going to ask what the best approach is for making changes to the Pull Request I've already submitted. There are a small number of changes I'd like to upload to my repository (switching off debugging, fixing some of those /home/pi references, making sensitivity settings remembered between sessions, and hopefully having a look at that error function in main.py). Should I simply update my repository with the changes, cancel the original pull request and create a new one from the latest commit once I'm ready? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @caracoluk for submitting your pull request, and for all the work you've put into this so far! A few points to hopefully clear things up:
In order to deploy this for real, we'd need to get it working with CustomPiOS and our GitHub workflow, as this is how we keep consistency with the OS setup. It sounds like there's a bit of work to get this to a deployable state. I suggest we merge this pull request into a new dev branch, where we can work together to make the necessary changes for deploying this. |
Beta Was this translation helpful? Give feedback.
-
|
Nice, I've seen this conflict too late, but anyway I couldn't act on it. We might want to add a line in the .gitignore to avoid this file to be tracked (naturewatch_camera_server/static/data/config.json). I was thinking about the computing budget. Is there any interest in offloading some task to the client (like the YUV420 to RGB conversion and the timestamp overlay)? The motion detection use a greyscale image anyway. Here we are converting the YUV420 image to an RGB image, then we convert the RGB image to greyscale (the greyscale image is already in the YUV image). |
Beta Was this translation helpful? Give feedback.
-
|
I did carry out a fair bit of performance testing in my attempts to boost the frame rate and I couldn't really detect much of a difference at all when I completely disabled the timestamp overlay process, and that was testing on a Pi Zero W. I must admit that I left the motion detection function completely untouched, but I suspect there are changes we can make there to improve performance. As you say, the Y channel of the YUV format is already greyscale so we should be able to avoid the colour conversion that's currently taking place and just extract the Y channel instead. I'll take a look at that once I've finished my current round of testing. I have both my Pi Zero 2Ws set up in debug mode at the moment to see whether they fail after the latest code update. So far neither has failed, but it's still too early to draw any conclusions. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know if you need it, but here is how I tackled the YUV to RGB conversion. I've added a function to only extract Y, I don't know if it's simply a matter of replacing it in the code. From what I've seen, the conversion is done for each captured frame before any call to import numpy as np
def yuv420p_to_yuv444(yuv420: np.ndarray) -> np.ndarray:
height, width = yuv420.shape[:2]
height = height // 3 * 2
y_size = width * height
uv_size = width * height // 4
yuv420 = yuv420.flatten()
# Extract Y, U, V planes
y = yuv420[0:y_size].reshape((height, width))
u = yuv420[y_size:y_size + uv_size].reshape((height // 2, width // 2))
v = yuv420[y_size + uv_size:].reshape((height // 2, width // 2))
# Upsample U and V planes
u = u.repeat(2, axis=0).repeat(2, axis=1)
v = v.repeat(2, axis=0).repeat(2, axis=1)
image_array_yuv = np.dstack((y, u, v))
return image_array_yuv
def yuv420p_to_grey(yuv420: np.ndarray) -> np.ndarray:
height, width = yuv420.shape[:2]
height = height // 3 * 2
y_size = width * height
yuv420 = yuv420.flatten()
# Extract Y, U, V planes
return yuv420[0:y_size].reshape((height, width))
# Example (you need a YUV image array)
# Display greyscale image using PIL
grey = yuv420p_to_grey(image_array_yuv)
# grey = grey.astype(np.uint8)
img = Image.fromarray(grey, 'L')
print(img) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @Vincent-Stragier. I was thinking of just extracting the Y channel using |
Beta Was this translation helpful? Give feedback.
-
|
I've modified the motion detection code so it no longer carries out the initial RGB to grayscale conversion, but instead strips the luminance channel directly from the YUV image. It should result in a performance gain. I think we're almost there now in terms of the conversion to picamera2/libcamera. The only remaining obstacle is the bug I've mentioned in the thread below which may or may not be related to my hardware. I'll continue my tests in that area, but as it can take about 4 days to fail, it's likely to be a couple of weeks before we have a complete picture of the problem. |
Beta Was this translation helpful? Give feedback.
-
|
I think I might have found an issue. On my Pi Zero 2W I've found that the circular buffer seems to hang every so often. Sometimes it can be after a few hours, other time it can go several days. As far as I can tell it's not a problem with the code and I'm wondering if it might be a bug in the kernel/camera drivers. When it happens, the web interface appears to work but the lores feed has frozen and attempts to call /api/settings also hangs. There is nothing in journalctl for the python.naturewatch.service of any interest, but shortly afterwards messages such as the following are logged in dmesg:
and I've been consistently ruling out possible causes, and have so far eliminated the following:
I'm currently testing a lower framerate of 15fps to see if that prevents the problem. I've attached a gdb python debugger to the code when it's in this state and the most recent call when it's hung is in the ChangeDetector class at the following point: That was originally a sleep of 0.02 seconds and I increased it to 0.03 to see if that helped since a frame rate of 20fps would allow for a sleep of 0.05 seconds and I wondered if perhaps we're asking for frames too quickly. So, after testing the lower frame-rate I'll increase this value again to see if that has any effect. In the meantime, I'd be grateful if others could try running the latest code and leave it running in video capture mode for at least several days as it would be good to know if it's an issue with my hardware. |
Beta Was this translation helpful? Give feedback.
-
|
@Vincent-Stragier The system itself remains operational. The service/Python process carries on running. No defunct processes are now shown since the last code update. The stream from the camera just stops and the image on the web page is frozen. When it's in that state, if I run |
Beta Was this translation helpful? Give feedback.
-
|
@caracoluk, I think it's more related to Picamera2 being in beta than to this repository's code. You should consider opening an issue in the Picamera2 repo to see if they have any thought about the issue. |
Beta Was this translation helpful? Give feedback.
-
|
I've thought about that but I'll need a way to easily reproduce the problem first. A simple few lines of codes that will give them something to work on. I'm not there yet, still narrowing down what's causing it. All the similar issues I've seen reported have resulted in requests for a simple way to reproduce the problem or they generally don't get much attention. Hopefully I'll have more information within the next week or so. |
Beta Was this translation helpful? Give feedback.
-
|
The Pi Zero 2W I have been testing to determine whether the app hangs if video capture has not been enabled has now been running over a week. This is significantly longer than it has ever run without hanging if video capture has been left running. So, I'm confident that the cause of the hangs is either in the code that runs with video capture enabled, or is caused by the increase in CPU load/temperature with that running. There are two things that are taking place once video captures are enabled. Firstly the H264 encoder is started, and secondly the motion detection functions are active. A couple of weeks ago I created a very simple Python process which left the encoder running with the exact same settings as the app. That ran for a week without hanging, so it would seem to eliminate that as the cause. So, at the moment it would seem to be caused by the motion detection functions. I suspected that might be the case, so midway through last week I left my other Pi Zero 2W running with an alternate motion detection function. This test has been running almost 4.5 days, which is about as long as the app has ever run without hanging with video captures enabled. It's still too early to say if this has fixed the problem, as I'd rather it be running at least a whole week before I'm confident. The new motion detection algorithm is a bit less intensive on the CPU, so it's still possible it's a CPU load issue. I very much doubt it's a temperature problem however as it ran quite happily on my window sill left in the Sun on the hottest day of the year so far when the CPU temperature reached 80C. I should point out that the motion detection algorithm I'm using works differently to the previous one. The new one compares the current frame with the previous frame, assigning a numerical value to the differences across the whole image. If the differences are above a threshold the capture takes place. I realise this isn't quite as sophisticated as the previous contour size checking that was taking place. However, I have modified the web interface so the the sensitivity is now adjusted using a slider with values 1-10 which does help offer better granularity. So far my testing has shown this works quite well, but I appreciate that it could lead to more false positive detections if there's lots of movement in the frame (e.g. the wind blowing leaves on trees etc.). Personally I think that's probably better than having the app hang every few days though. I'll update my repository in a moment, and will provide an update by the end of the week to confirm whether the issue with the hangs has truly been resolved. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately both of my Pi Zeros hung again earlier today. At this point I'm pretty much out of ideas as to what's causing the issue. It's possible it's a bug in the encoder, and my test script just happened to go a week without exhibiting the problem, but I can't be sure. With no simple way to reproduce the problem I can't see this being easily fixed. I've put together a watchdog script which will restart the service and re-enable captures should it hang (watchdog.sh in the helpers directory) but at the moment I think this is the best I can come up with. |
Beta Was this translation helpful? Give feedback.
-
|
I was carrying out some performance testing over the weekend, attempting to obtain better video frame rates. I explored setting NoiseReductionMode in PiCamera2 to either "Off" or "Fast" and also changing the H264 encoder quality from "Quality.HIGH" to "Quality.MEDIUM". These changes made negligible difference to frame rate so I didn't change them. I also tested writing the initial H264 file to a memory buffer rather than straight to the SD card. This didn't really help either so I left it saving straight to SD card. What did make a very noticeable improvement was setting the video output format to YUV420 instead of RGB888. With that change the Pi Zero 2W will happily run at 25fps, most likely higher but on my Pi that pushes the CPU temperature above 80C so I'm leaving the default at 20fps for the time being and it can be changed in the config.json file if necessary. I've also added a check in ChangeDetector.py as there's no point in running the motion detection function if a video is in the process of being saved. Saving those CPU cycles does help push up the frame rate by reducing the number of dropped frames. In a similar fashion, the same check is being performed in CameraController.py so if a video is being recorded the video stream on the web page is only updated once a second, instead of every 0.03s. Once the recording is complete the values are returned back to 0.03s. I think performance is no longer an issue on a Pi Zero 2W, but even with these changes I couldn't quite get 15fps out of my Pi Zero W. It can just about manage 12fps, but any more than that and frames are dropped. Once the motion is detected, and the motion detection routine is temporarily disabled, the Pi Zero W can manage 15fps, but the problem is the contents of the 5s buffer before motion is detected. While that was being filled with motion detection enabled, the Pi Zero W can't write frames at 15fps, only managing 10-12fps. This means that it takes longer than 5s buffer to fill the buffer. When the buffer is written to the SD card, instead of containing 5s of data, it has about several extra seconds. This is a problem because ffmpeg expects a consistent 15fps when it converts the H264 file to an mp4. The end result is that the MP4 file contains about 19s of content instead of 15s. So, those 19s play back in 15s which obviously doesn't look correct. Perhaps we should simply release the new web app wit the caveat that it's designed for the Pi Zero 2W, though will work on the Pi Zero if the frame rate is set to 10-12fps. I have this morning committed these changes. |
Beta Was this translation helpful? Give feedback.
-
|
This morning I've updated the watchdog script (/helpers/watchdog.sh) so that it now checks every 5 minutes if the service is responding properly. Each time the check is run it monitors the state of the capture mode (inactive/photo/video/timelapse). If the service stops responding, it restarts and then re-enables the last capture mode detected. I've updated the installer scripts to schedule the watchdog to run at each boot. I don't like having to rely on a watchdog script, but after spending weeks trying to work out why the service hangs intermittently after a few days I feel like it's time to settle for this approach instead. I'll leave my two Pi Zero 2W running for at least the next few weeks, but I think we at least now have something that can be considered for deployment. |
Beta Was this translation helpful? Give feedback.
-
|
Now that both of my Pi Zero 2Ws have been left running for 10 days I'm happy to say that both have remained running. One Pi has remained running for the full 10 days without the service hanging at all. That's further confirmation that the recent code change has made things much more stable. The app hung twice on the other Pi over the course of the 10 days, but on each occasion the watchdog detected the hang, restarted the service and restarted the video captures. Therefore I'm happy the code is now stable. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, what's the current status please? Is it possible to try out the latest version to support the Pi Camera 3 on a Pi Zero2W please? |
Beta Was this translation helpful? Give feedback.
-
|
It’s in a fully working state and the code is in a finished state. There’s a firmware bug that is partially fixed by applying the latest Pi kernel update, but since there is a watchdog automatically restarting the service if it fails that should not matter. I’ve been running the latest code on two of my Pi zero 2Ws for the last few months and they’re running fine.
The only thing to keep in mind is that the latest code is not packaged up for distribution yet so you would need to follow the steps described in the comments above to set it up yourself.
|
Beta Was this translation helpful? Give feedback.
-
Hi everyone,
We're looking for people to help us transition My Naturewatch to picamera2 and libcamera. Unfortunately we don't have funding to work on this project actively, but do have a little bit of time to do code reviews and coordinate this. We are looking for contributors who are willing to give some of their time to transition the codebase to picamera2.
In particular, the CameraController class will need to work with picamera2. Photos should be easy, but video may be a challenge, as we use circular recording to capture the action before the camera was triggered.
Let me know if I can help in any way. To our lovely contributors so far (@egueli, @frak, @raphaelcoeffic, @HAZET, @Vincent-Stragier, @MrChaz, @Gadgetoid), thank you for all your contributions, and I hope some of you may have some time to donate to this. We and the My Naturewatch Camera community will be so grateful!
Beta Was this translation helpful? Give feedback.
All reactions