Last Active: Jan 05, 2019
Threads: 10
Posts: 230
Reputation:
5
Jun 06, 2018, 02:39 am
(This post was last modified: Jun 06, 2018, 02:42 am by jerrysmatrix. Edited 1 time in total.)
I think that it would be beneficial to use SQLite for this project. The initial import process would take a bit longer, however searching would be much faster, and it would open up possibilities for some new powerful features. I thought about working to implement this, but I wanted to bring it up to see if that's something you would be interested in.
Edit:
Also please bear with me; I've been a bit of a lone coder up to this point. I do not have much experience in collaborative work with GIT, so I was planning to use this project to gain some experience with that.
Last Active: Jul 25, 2022
Threads: 3
Posts: 58
Reputation:
2
(Jun 06, 2018, 02:39 am)jerrysmatrix Wrote: I think that it would be beneficial to use SQLite for this project. The initial import process would take a bit longer, however searching would be much faster, and it would open up possibilities for some new powerful features. I thought about working to implement this, but I wanted to bring it up to see if that's something you would be interested in.
I've tested with SQLite. No difference at all. In fact the search process is slower by about 2 seconds. Even with CLI. Papaparse is a monster when it comes to CSV. Another thing is that all the SQLite modules are natives which will create a whole new world of problems for Electron. Even if we add SQLite support we still need the CSV parser. It's not worth adding a huge extra dependency. Also it'll immediately take out the ability to cross-build. Users will also need platform build tools. It took a lot of effort to avoid native modules. I think it's better if we keep it that way. A lot of electron devs follow this rule to avoid a lot of complexity.
Last Active: Jan 05, 2019
Threads: 10
Posts: 230
Reputation:
5
Jun 06, 2018, 03:35 am
(This post was last modified: Jun 06, 2018, 03:38 am by jerrysmatrix. Edited 2 times in total.)
(Jun 06, 2018, 03:07 am)techtac Wrote: I've tested with SQLite. No difference at all. In fact the search process is slower by about 2 seconds. Even with CLI. Papaparse is a monster when it comes to CSV. Another thing is that all the SQLite modules are natives which will create a whole new world of problems for Electron. Even if we add SQLite support we still need the CSV parser. It's not worth adding a huge extra dependency. Also it'll immediately take out the ability to cross-build. Users will also need platform build tools. It took a lot of effort to avoid native modules. I think it's better if we keep it that way. A lot of electron devs follow this rule to avoid a lot of complexity.
Wow that's surprising. I thought for sure SQLite would've been faster. I mean in theory it definitely should be, but I guess I'm a bit out of touch nowadays.
Just out of curiosity, were you using a SSD when you tested it?
Last Active: Jul 25, 2022
Threads: 3
Posts: 58
Reputation:
2
(Jun 06, 2018, 03:35 am)jerrysmatrix Wrote: Wow that's surprising. I thought for sure SQLite would've been faster. I mean in theory it definitely should be, but I guess I'm a bit out of touch nowadays.
Just out of curiosity, were you using a SSD when you tested it?
I used both HDD and SSD. Doesn't do much of a difference. I'm using the SSD on the video. It may have something to do with the buffer allocation.
On a second note, I wanna optimize the NeDB functions with dot notation. It was a disaster on the last attempt because it seems like NeDB is not very good with dot notation. Slowed down the process. Can you try to reproduce this issue? We'll do a comparison. My guess is current callback hell performs better than the dot notation optimized version.
Last Active: Jan 05, 2019
Threads: 10
Posts: 230
Reputation:
5
I would do a benchmark for this, but I'd rather have some more details on your test. What method did you use to benchmark it? Was it just a few simple strategically placed timestamps? Did you already have a test written? Did you benchmark papaparse or NeDB, and was it an isolated test or did you test on of your files?
With V8, dot notation should be slightly faster than brackets. The difference is very minimal, but there is definitely something wrong if dot notation is coming up slower than brackets. If there's a problem like this, my guess would be that one of these NPMs/libraries is supplying objects with getters/setters, or some other weird mutated objects.
Last Active: Jul 25, 2022
Threads: 3
Posts: 58
Reputation:
2
Jun 07, 2018, 04:25 am
(This post was last modified: Jun 07, 2018, 04:26 am by techtac. Edited 1 time in total.)
(Jun 07, 2018, 04:11 am)jerrysmatrix Wrote: I would do a benchmark for this, but I'd rather have some more details on your test. What method did you use to benchmark it? Was it just a few simple strategically placed timestamps? Did you already have a test written? Did you benchmark papaparse or NeDB, and was it an isolated test or did you test on of your files?
With V8, dot notation should be slightly faster than brackets. The difference is very minimal, but there is definitely something wrong if dot notation is coming up slower than brackets. If there's a problem like this, my guess would be that one of these NPMs/libraries is supplying objects with getters/setters, or some other weird mutated objects.
There aren't any unit testings. But i don't remember exactly if i committed it or not. I tested CLI on as a completely different project. Please go through the commits once. I should've branched this. But i wanted to stick to linear so badly.
I know dot notation should be faster. That's why i was frustrated with this. This is on NeDB's side. NeDB has got some major issues. For one, no muti-process support. There's no fsync mechanism or queuing in NeDB. If you're good with unit-testing feel free to add some. I'm one of those guys who find unit testings a waste of time. But i'm not ignorant. I know they're structural and very important for a lot of people. That's why people like us collaborating is good. 
Last Active: Jan 05, 2019
Threads: 10
Posts: 230
Reputation:
5
Honestly, I have little experience with unit testing. I'm good at pinpointing bugs and bottlenecks, but I've always done it the hard way. This might the type of practice I need, but first I'll check around when I get some time, to see if anyone else has written some tests already. I'm trying to get into the habit of not reinventing the wheel for everything.
Last Active: Jul 25, 2022
Threads: 3
Posts: 58
Reputation:
2
(Jun 07, 2018, 04:56 am)jerrysmatrix Wrote: Honestly, I have little experience with unit testing. I'm good at pinpointing bugs and bottlenecks, but I've always done it the hard way. This might the type of practice I need, but first I'll check around when I get some time, to see if anyone else has written some tests already. I'm trying to get into the habit of not reinventing the wheel for everything.
Haha. Ditto.
Last Active: Jun 07, 2018
Threads: 0
Posts: 1
Reputation:
0
When I open the .exe file, nothing happens, and I've even tried opening with Admin privileges...idk what to do.
Last Active: Jul 25, 2022
Threads: 3
Posts: 58
Reputation:
2
(Jun 07, 2018, 16:02 pm)f15h3gg5 Wrote: When I open the .exe file, nothing happens, and I've even tried opening with Admin privileges...idk what to do.
Saw an open issue on the repo. Lemme check.
|
