An Old DBA Learns AI: I Tried to Train an LLM for the First Time

The Goal

The idea was simple.

I wanted to take a general-purpose LLM and give it a small piece of DBA knowledge.

For the first experiment, I deliberately kept things extremely small.

I used one Oracle DBA question:

That's it. One question. One answer.

This wasn't intended to create a production DBA model. It was an experiment to understand what actually happens when you fine-tune an LLM.

The Hardware

I used Google Colab with a Tesla T4.

The environment reported:

The GPU had 15 GB of physical memory, with approximately 14.56 GB available to PyTorch.

That was enough to experiment with a 7B model using quantization and QLoRA.

The Model

I used:

The model has roughly 7.6 billion parameters.

A full fine-tuning run would be much more demanding than what I wanted to do on a Tesla T4. That's where QLoRA comes in.

What Is QLoRA?

The simple explanation:

Instead of updating all 7.6 billion parameters, we keep the original model largely frozen and train a much smaller set of additional parameters using LoRA.

The model was configured with:

And this was the interesting part. The training output showed:

So I wasn't training the entire 7.6 billion parameter model. I was training approximately 5 million parameters. That's only 0.0662% of the model.

For me, this was one of the most interesting moments of the experiment.

Preparing the Training Data

I started with a single training example:

I converted it into a Hugging Face Dataset:

Then I converted the example into Qwen's chat format:

The final example contained 64 tokens. Then it was converted into input_ids, attention_mask, and labels.

At this point the data was ready for training.

My First Training Run

I configured a very small experiment:

Then I created the Hugging Face Trainer and ran train_result = trainer.train().

This was the first moment where the model was actually being trained.

And It Worked...

Technically. The training completed all three epochs. The training loss was:

So the loss went from 4.63 → 3.82. That was exciting to see.

My first thought was: "It worked. The model is learning." But then I tested the model. And that's where things became much more interesting.

The First Problem

My first post-training generation produced a strange repetitive output:

Obviously, this wasn't a useful DBA answer. There were also warnings related to gradient checkpointing and use_cache.

So I switched the model to inference mode, disabled gradient checkpointing and tested it again. This time the output looked much more normal. The model produced a long Oracle troubleshooting answer.

But there was another problem: The answer wasn't necessarily correct.

Lower Loss Doesn't Mean Better DBA Knowledge

The model produced an Oracle-style troubleshooting answer, but some of its recommendations were questionable. For example, it focused heavily on locks and long-running transactions and suggested approaches that weren't necessarily appropriate for ORA-01555 troubleshooting.

So I decided not to stop there. I asked a different question:

"What are the main causes of ORA-01555 in Oracle Database?"

The model again produced a detailed Oracle-looking answer talking about long-running transactions, concurrency, row versioning, read consistency, and isolation levels.

But it also introduced questionable information, including a reference to a DBSNMP user's DB_VERSION_RETENTION parameter. That wasn't the kind of reliable DBA knowledge I was hoping to get.

The Final Test

I wanted to see whether the model could distinguish between two completely different Oracle errors. So I asked:

"What is the difference between ORA-01555 and ORA-00600?"

The model correctly described ORA-00600 as an internal Oracle error. But its explanation of ORA-01555 was wrong: It claimed that ORA-01555 was related to a table or index being altered, dropped or renamed during a transaction!

That was a clear factual problem.

The Model Had Been Trained — But It Hadn't Become a DBA

The technical training pipeline worked. The QLoRA adapters were trained. The loss decreased. The model changed its behavior. But that didn't mean it had learned reliable DBA knowledge.

My experiment was based on: One question + one answer. That's nowhere near enough data to teach an LLM a complex domain like Oracle Database administration.

What Did I Actually Learn?

This experiment changed the way I think about fine-tuning. Before this experiment, I was thinking about fine-tuning mainly as:

But the real process is much closer to:

The training data matters enormously. If the training dataset is tiny, incomplete or contains incorrect information, fine-tuning won't magically fix that. It can actually make the model behave differently without making it more trustworthy.

Fine-Tuning vs RAG

This experiment also made the difference between my previous RAG experiment and fine-tuning much clearer.

With RAG, the idea is roughly:

The knowledge lives outside the model.

With fine-tuning, we're changing the model's behavior through training:

That distinction became much more real to me after actually running the experiment.

Was the Experiment a Failure?

I don't think so. The technical experiment succeeded. I managed to:

• load a 7B LLM on a Tesla T4
• use quantization
• configure QLoRA
• train only 0.0662% of the model parameters
• complete a training run
• observe the training loss decreasing
• generate responses after training
• test the model on questions it had never seen

But the knowledge experiment was not successful. And that's actually more interesting, because I now have a much better understanding of what I need to do next.

What Would I Do Differently?

The obvious next step is to increase the quality and size of the training dataset.

Instead of 1 question + 1 answer, I would build a proper DBA dataset containing examples around:

And, most importantly, I would need to validate the answers before using them for training.

Then I could compare: Base model vs RAG vs Fine-Tuned model using the same evaluation questions. That would be a much more meaningful experiment.

Tech Stack