I agree that OP sounds like a beginner, and what you’ve suggested is likely the best approach for someone who is familiar with frontend tools and frameworks. Selenium (and admittedly BeautifulSoup) is probably too low level for this particular user, but that doesn’t mean they can’t still learn some fundamentals while solving this problem without resorting to something as heavy and complicated as background browser emulation and rendering. I could be wrong though.
I’m not currently on Discord, could you upload the code to pastebin or something similar?
I would love to see your code, but I understand if this forum isn’t the most ideal place to share.
In my experience, this scenario typically means that there is some sort of API (very likely undocumented) that is being used on the backend. That requires a bit more investigation and testing with browser developer tools, the JS Console, and often trial and error. But once you overcome that (admittedly very complex and technical) hurdle, you can almost always get away with just using the requests library at that point.
I’ve had to do that kind of thing more times than I’d like to admit, but the juice is almost always worth the squeeze.
Selenium is really more of a testing framework for frontend developers, and could theoretically be used for scraping, but that would be somewhat like buying a car based on the paint and not looking in detail under the hood.
I can’t say I’ve ever worked with scrappy, but the tool I would use for web scraping with Python is BeautifulSoup. This tutorial seems decent enough, but you will need to understand basic web concepts like IDs, classes, tags, and tag attributes to get the most out of the tutorial: https://geekpython.medium.com/web-scraping-in-python-using-beautifulsoup-3207c038723b
W3Schools will also be your friend if you have questions about HTML/CSS selectors in general: https://www.w3schools.com/html/default.asp
Understanding regular expressions and/or xpath would also be very helpful, but are probably best considered to be extra credit in most cases.
I’ll try to respond if you have any issues or questions, but hopefully that gives you enough to get started.
Been a little while since I worked on ODBC stuff, but I have a couple of thoughts:
Would it be possible to use something like a table function on the DB side to simplify the query from the ODBC side?
I could be misremembering, but I feel like looping through individual inserts with an open connection was faster than trying to submit data in bulk when inserting that much data in one shot. Might be worth doing a benchmark in a test DB and table to confirm.
I know I was able to insert more than 50M rows in a manner of single digit hours, but unfortunately don’t have access to that codebase anymore to double check the specifics.
Thank you for the detailed response!
I see now that you were responding to the title and not the content of the post.
To further round out my knowledge and understanding, could you please point me to additional information about Latinos moving further to the right, Harris’ popularity, and polling data showing her well behind Trump in key states?
I’m also curious to see examples of the American left refusing to acknowledge the potential of a second Trump term.
Thanks again for helping me understand.
I’m not sure if I understand the points you are trying to make. Could you please elaborate? A) this meme is about records broken, not “wins.” A president winning a second term would not break any records. B) why wouldn’t Americans vote for a black woman? C) why would the thought of Americans electing a black women be “cult like?”
I’m sorry, I hope I didn’t offend you.
I’m still not sure if I understand the intended joke. An average of 41 attempts per person sounds horrific. I’m sure there is something that’s going over my head, it’s some kind of dark humor, correct?
I sincerely hope that she has been trending in a positive direction. I’m glad to hear that her attempts have been unsuccessful, and that she has good love, support, and (hopefully helpful) medication in her life.
I imagine the knowledge of your sister struggling and suffering is hard on you too, and I wish the best for both of you.
Thank you, I agree that it’s an important distinction to make. Having only been able to read the abstract of the linked article, do you perhaps have any information on the number of completed attempts compared to unsuccessful attempts?
I hope it goes without saying that even a single attempt is too many, and any completed attempts are devastating tragedies that don’t reflect kindly on our current society.
I don’t understand, could you please elaborate?
x = x/1 = x is true for all numbers, no?
Thank you, that is absolutely heart-wrenching.
Is that referring to 41%, 41 per 1000, or some other metric entirely? Either way it’s too damn high, but I’m curious and I don’t know how to go about researching it without potentially making the almighty algorithm think I’m anti-trans.
Rizz is Kah-RIZZ-mah (charisma). Usually used in the context of one person hitting on another with reciprocated attraction.
Automation. My phone automatically triggers API calls, settings tweaks, launches apps to specific pages/playlists, and collects usage statistics to a local and private location all on its own. This means I only get a day and a half of battery life, but the tradeoff is well worth it in my opinion.
https://mashable.com/article/iphone-2g-original-naysayers
https://forums.anandtech.com/threads/saw-this-on-reddit-tonight-a-2007-iphone-will-fail-column.2240010/ (can’t seem to find the original article, so this is commentary and quotes from what the article said at the time)
https://news.ycombinator.com/item?id=7419784 (additional quotes and commentary on the same Bloomberg article, with other similar articles mentioned)
http://suckbusters2.blogspot.com/2007/06/apple-iphone-debut-to-flop-product-to.html?m=1 (good summary of the points that were made in opposition)
Android was going to be very different before the iPhone was revealed. That moment very clearly ushered in a new era for smartphones and phone technology in general. Now smartphones weren’t just for business people with an IT team for support, or wealthier and tech savvy individuals, they were for the masses. That undeniably changed the world (for better or worse), and there’s no way anyone could have predicted the outcome with any confidence at the time.
Perhaps Vision Pro will be a success and usher in a new revolution in technology, or perhaps it will only ever be a niche device, or maybe it flops altogether in a short amount of time. No one can say for sure how things will play out until it happens.
What if the person with the Vision Pro got it to help with a physical disability, and it has greatly improved their quality of life? Or what if it was gifted to them by a now passed friend or family member, and now holds a great deal of sentimental value to that individual? Do you not agree that criticizing in these (and likely many other) instances would be an asshole move on your part?
You don’t know how or why they obtained it, and their possession of it does not harm you in any way, shape, or form. Do you still not agree that being vocally and directly critical of the other person’s simple possession of this item is an asshole move?
Why did they submit this as a pull request in the first place? Just commit it to a WIP branch until it’s ready to merge. Am I missing something?
100% this. Every website is different, though after doing this kind of thing for long enough, there are often common patterns and frameworks/libraries. Even general obfuscation can be reasonably reverse engineered with enough time and effort.