Screen scraping with AI

I feel like I am very late to the party. Again.

This time it is amazing AI-party where almost everyone is invited – some more than others – but I had let the party go on for quite some time before I decided to have a peak inside.

It started when I had to do a test for an AI startup which I had applied for a position with. This of course leaves the attentive reader with a very good question:

Why are you applying for a position with an AI startup if you do not have experience with AI?

But the even better question might be:

Why has the AI startup decided to even let you get as far as to a paid test, when you clearly does not have AI experience on your CV?

I can answer the first question clearly but for the second one I can only speculate. I applied because first of all I wanted to try and gain some experience in this area. I have worked a lot as freelance Enterprise Architect, Principal Solutions Architect and Product owner even though my background is as developer. But in doing so I have been missing the deep work of coding. And the job posting this company had put up required experience with React and they would prefer if the candidate had prior experience with managing full projects themselves. AI knowledge was not required. As I have co-founded two startups myself which I coded and managed I qualified for this part.

On top of that they apparently liked my cover letter and CV enough to give me a chance.

The test and accidentally realizing AI was part of it

The test was to be conducted in a 4 hour block that I should complete within a weeks time whenever I felt like it.

First of all I was a little nervous. I has been many years since I had done anything even close to such a test. I have coded in my spare time and played around with different projects, but not regularly and not recently.

When I first got the test I was overwhelmed. It seemed like too big of a task and I used a lot of energy trying to figure out where to start. Reading through the guidance a few times helped. At least I was calm I knew that I should not rush the start as I could end up wasting time going in a wrong direction. Better be sure what I want to build before starting.

But I had to admit that it took a number of reads through the assignment before I found a good way forward. They mentioned some files that could be interesting to look at and one of them presented a page where you could ask an AI agent to create a website. So you had an input field and an array of “pre-configured” website examples it could build. So I tried one of them and realized it could spit out quite substantial and well working code. So finally the light went on in my head.

I could ask the AI to generate a lot of the required code from the task.

So on I went with my very first attempts at prompting AI agents for code. I was impressed with the results I could get as long as I asked the right questions. Rather quickly I had generated an example that got me close enough – I had to be vary of the deadline and not spent all my time on this part of the assignment – but I ran into another issue as the code base I had to integrate into used Typescript and I had zero knowledge of Typescript. The issue was that the code generated did not live up to all the Typescript-related validations and even though I tried a number of tries at generating more compliant output from the AI agent I finally had to throw in the towel and just work through them one by one wasting an enormous amount of time in doing so.

When I finally had working code I could only make the most basic integration into the code base and overall I came away with an experience of being in way over my head.

Pandora’s box

But as the days went I kept the experience somewhere in the back of my head until I finally this morning had the idea of using AI to help with an old idea and project I have had lying around for a few years.

This idea involves quite a bit of screen scraping and writing custom scrapers for a number of different sites that I need to keep check of whether are updated or not. So this morning I went ahead and gave it a try with what ended up being three(four) different AI agents.

ChatGPT

First I tried with ChatGPT and it quickly turned out a C# class with the code needed to fetch data from the site, build a little data model and fill it with information from the site. Great start!

But as soon as I had integrated the code into my solution and tried to run it I found issues. It looked fine at the outset but the selectors for the XML were off and it missed a number of properties that could have been nice to have in the data model. Of course all doable fixes, but as I had other AI solutions open in the browser I might as well try their take.

Microsoft Co-pilot

First I tried the GPT 4.1 model and to be fair I cannot really remember how impressed I was with the result. It must have been close to the ChatGPT answer as I did not use it but instead tried using Co-pilot with Claude 3.5 Sonnet model and this gave a really good answer back. At least better than the others so far. It had more properties about the individual parts I wanted to get from the website and the XML selectors worked better out of the box than the ones from ChatGPT for instance. It was by no means perfect but it was close enough to have something to work with.

Interesting enough it also had split out some of the functionality into sub-methods whereas the others kept it inside a single method. This can of source be explained by the fact that Claude 3.5 Sonnet also found more information to fetch and hence perhaps broke down the code into smaller blocks. The others may not have done so as there was only 4-5 properties in their “data-model”.

This code-snippet also included some error-handling which the others did not.

Perplexity.ai

Finally I tried perplexity. Mostly just as a test as I at this point liked the starting point Claude 3.5 Sonnet had given me. Perplexity seemed to land close to ChatGPT. So not great. but by all means a good starting point to work from.

Key takeaways

First of all I am a little impressed with all of these models abilities to churn out code. Of course this is not the biggest surprise in the World but still impressive to see. This was not the biggest or most complicated test for them, but they still deliver results that could work as a good starting point. I think the surprise for me in this case was more in how much work there needed to be done afterwards to have a full working screen scraper. I am not wise enough to know exactly how they work, but if they look at the page itself they have access to all the XML and should in my opinion be able to write fully functional code without the need for manually going through the selectors afterwards and correct them to actually fetch the information the tried to get in the first place.

I can see the power of an integrated AI model that can see the context you are working on as a developer. Ask it to write some boilerplate code to create data- or business entities, connect these write code to consume an endpoint. If you become really good and efficient at this then you can really cover some ground as developer and not waste time on trivial tasks.

But what is also apparent is that you need to be a good developer to evaluate the code you get back. You cannot just take it at face value. You need to be able to use it as a starting point and work from there. In cases where you are lost you may get a step in the right direction. The only potential downside here is that if you are working on something you know very little about, then it becomes harder to evaluate whether the answer you have received is good or not.

Then again, this has always been the case with developers. We have always looked for solutions and found code snippets on Stackoverflow and others places only then having to integrate and try to understand it afterwards.

So I think my main takeaway is that with the right model, the right developer and the right project then AI-fueled development is really night and day. I can see it help me create prototypes that would have taken me months in only a matter of hours. Then there would be a big task of correcting, fine tuning etc. but I could cover a lot of ground really quickly.

For production-ready code integrated into a clean architecture I am a little more hesitant. But perhaps that is just because I am not good enough at using it yet or that I have only had access to the free “old” models and the new ones would fix the issues I have encountered.

Anyway, the most important thing that came out of this experiment was that I moved a step in the right direction with a project I have had on the shelf for a number of years. Perhaps I will now finally be able to complete it. At least I have an easier way to build a good starting point for the individual pages I want to fetch information from.