Whatpulse 2.0 saving to db file every second?

The title says it all. Why oh why is the client saving the data to the sqlite database every friggin second?

That’s a <4kb write every second. That is going to kill peoples’ SSDs.
And for the people that are still using HDDs it means that their harddrives will always be in full operational mode and never ever start to use their energy saving features.

This is completely unacceptable behavior. If that won’t change, I’m no longer using it. I can think of way more reasonable and useful ways to kill my SSD.

Well, yes and no.

An SSD can write to a page at a time - a page is typically 4 kB. Though when the SSD has to delete data, it has to delete a block (512kB) at a time. When Whatpulse writes 4kB at a time, it writes to a page. When it needs another 4kB, it writes to another page, and makes the previous page ‘invalid’, so the SSD can garbage it later. When the block is full, it will just move onto another block, until the whole drive is full of invalid pages. Then it’ll clean them up (garbage collection or other) and start again.

http://www.anandtech.com/show/2738/5

Every 4kB write makes an SSD write a whole page of data. Even with wear levelling, write amplification and spare area, those 3000-20000 P/E cycles (depending on the drive) will not wear down quite quick. Let’s do the math…

Assume 128GB SSD (including spare area), empty.
Each 4kB write requires a whole to be written
The way an SSD works is that if 4 kB is free in a block, it will write only the new page and ‘lose’ the address to the old data.

A 128GB drive will have
128 GB / 4 kB = 33554432 pages
Which means (33554432 / 3600) = 9320 hours to go through one entire P/E cycle

@ 3000 P/E cycles = 873813 hours = 3192 years
@ 5000 P/E cycles = 5320 years
@ 10000 P/E cycles = 10640 years

While I agree with you that having continuous writing could be bad for an SSD, it is not as bad as you think. It is when the drive has to delete and rewrite a whole block of data when the drive gets full which really screws it up.

Hence why Anand + Kristian @ Anandtech always suggest leaving 10-20% free space on your SSD.

True, in comparison to how many P/E cycles you have in a modern SSD the amount it will write is small. It won’t outright kill the SSD but still consume a sizable amount of writes.
In one of his articles Anand mentioned that he averaged 7GB of writes a day and that he considered himself an above average user.

With a 4kb write every second and let’s assume a write amplification of 2 for such small writes:

4kb * 3600* 24 * 2 = 691200 kb = 675 MB

That’s almost 10 % of Anand’s workload. And that’s just for the log of Whatpulse. That’s a bit excessive.
The problem with standard HDDs that I pointed to also still remains.
Basically with a write every second, the drive heads will never be parked and the drive will also never have the possibility to shut down entirely. In Desktop systems that’s not ideal but also not that big of a problem. On notebooks on the other hand this will not only impact battery life by a noticable amount, it will also make the harddrive a lot more vulnerable to shocks, because the drive heads will always be positioned above the platters.

And that’s for basically no good reason. At least I can’t think of a reasonable excuse as to why Whatpulse needs to persist every datapoint it collects directly and immediatly into the database. Even Windows’ own logfiles don’t use such a high frequency.
It just seems excessive to almost borderline paranoid to persist every second.

I’m pretty sure Anand quotes in his recent articles that ~10GB writes a day for an ‘average’ user, but it’s nothing for a power user.

Perhaps I should mention Anand is my direct boss :smiley:

I do wholeheartedly agree that it is an issue though, especially when you put mechanical drives or mobile devices in the mix.

I suspect that it is writing every second to update the ‘uptime’ component. Realistically having it to the nearest second is pretty pointless. Once a minute is more than enough.

First of all: Man I wish I was working for Anand :smiley:

Recording the uptime of the system and the applications running on it might be the culprit here, but that could just as easily be saved out on exit of the application. Or since the Application activity page seems to work in hourly increments, once per hour.

Well I got curious so i peeked into the db file and boy does this track a lot of data points. They basically save out all the applications, all the network activity (there’s even a table network_interface_bandwidth_second and another one “application_bandwith_second”), all the keystroke frequencies (duh) all clicks and all applications and their uptimes. Let’s just hope that EA never gets their hands on this code or Origin will turn into Skynet in no time :wink:

Then there is the “mousepoints” table. It does exactly what it sounds like. It contains the position of the mouse pointer at any given time. Sure that needs to be recorded in order to generate the mouse-pointer heatmap. But over a few hours that table alone has accumulated 2206 entries.

In fact with that many tables and that many datapoints the db file will most likely grow quite big over the months. Over the past 5 hours it has grown by a 120 kb. And that’s even though I didn’t install the network monitoring component. :wink:

I’m quite concerned about this. The client does indeed seem to perform an awful lot of writes per unit of time. I wouldn’t want this to wear down my SSD while I can’t think of a reason for not buffering the collected data in RAM before writing to disk. Could one of the developers comment on this? I’d like to be reassured it’s either not harmful or absolutely necessary and worth it :wink:

In the meantime I’ll downgrade to the final 1.x version… oh wait, that’s not allowed due to some kind of “security reason”. In my opinion that’s a pretty lame move since with the release of a major rewrite it’s to be expected people will run into issues.

I’m guessing that’s because you can run the 2.0 client and an 1.x one at the same time, so that security measure prevents you from being able to effectively use both on different profiles and double your key and click count.

I didn’t have much to add to borandi’s post. It’s really not a big issue, just a mindset. Would you rather have a big burst of data, or a steady stream?

You can restore pulsing for 1.x clients by recreating the computers you use with 2.0, so the ‘2.0’ tag disappears from your computers and the 1.x pulse process doesn’t see any 2.0 clients in your account.

All right, thank you for sharing your views. I’ll consider the information borandi posted. Thanks for creating the client by the way, it has some pretty cool new features. I’m by no means ungrateful, just trying to be protective of my rig :slight_smile:

Ah, that’s perfectly fine then. Perhaps you should re-phrase the notice on the download page a bit to reflect this, since it pretty much suggests it’s a point of no return.

How exactly do you recreate the computers?

Cheers,

Pete

Delete and create them from http://whatpulse.org/my/#computers

If it’s possible to downgrade back to 1.x Whatpulse, where would I be able to download a copy of the relevant WP 1.x versions for all three OSes?

With new version I’ve got another problem. Every second write under Linux cause journal indexing on ext4 partition. I have very slow and noisy hdd, and writing every second can be frustrating.
Is there a way to make and option in configuration and manually set this time? 1 second could be as default settings and can be changed to whatever you want, even 1 hour or more.

Since upgrading to 2.0 (and 2.0.1), I’ve noticed the disk accessing on a continuous stream (every second could be accurate), and even while the computer is asleep (it’s a 2011 iMac) I hear the ticking.

I’d rather give up a chunk of RAM or something than having the constant access.

Great upgrade, but this could be a deal breaker for me.

Thank you.

+1 for allowing user to set interval of db file saving.

Can any developer fix this as soon as possible, this is very annoying to hear the writing sound of my harddisk when my computer is not even used at all.

Yeah, would be great if if you guys could fix this. It’s very annoying and imho unnecessary to actually write this often.

This has been fixed in 2.0.2 :slight_smile: