How to Pick a Local AI Model for Writing Fiction

By Austin Little

You do not need the "best" model. You need the one that helps your sentences without stealing them, runs on the machine you already own, and keeps your unfinished book on your own disk. This is how to find it in one evening, for free.

AI disclosure. This page was drafted with AI assistance and edited for Apiary. Facts that need a live check are marked. We don't invent quotes, stats, people, or events.

Why "best model" lists don't help fiction writers

Most of the AI rankings you find online are built for people who want correct answers. They test math problems, coding puzzles, trivia, and reasoning chains. Those are real skills, and they matter for real jobs. But fiction is not a test with a correct answer. A story lives or dies on things those rankings barely measure: whether a character sounds like one person and not a committee, whether a scene has tension, whether the prose has rhythm, and whether the model can stay out of the way when you want it to.

A model that scores well on a reasoning test can still write every villain like a corporate apology. A model that scores lower can still be a better brainstorming partner for a mystery novel because it is willing to suggest strange, specific details. You will not find that out from a leaderboard. You find it out by testing the model on your own kind of writing.

So this guide is not a list of winners. Model names change fast, and anything I printed as "the best fiction model" would be stale before you finished reading it. Instead, this is a method. You pick a few free candidates, run them through the same short set of fiction tasks, score them with a simple sheet, and keep the one that helps you. It takes about an evening. It costs nothing but electricity.

If you want the hardware side of this question (what fits in a modest laptop), Apiary has a companion piece, Which Local Model for Writing on 8GB RAM, and another on running Qwen or Mistral locally for blog drafts. This article is the fiction-craft side: how to judge a model as a creative partner once you know what your machine can hold.

First, decide what job the model is doing

Before you download anything, write down what you want help with. "Help me write my novel" is too vague to test. Fiction help splits into very different jobs, and a model can be good at one and bad at another.

Jobs where local models often help

Brainstorming. "Give me twelve possible reasons the lighthouse keeper lied." You want quantity, variety, and a few strange ideas you would not have had alone.
Continuity checking. "Here is chapter three and chapter nine. Did I change the color of her car?" You want careful reading, not creativity.
Reverse outlining. "Summarize what happens in each scene of this chapter in one line." This helps you see structure. (There is a full companion on outlining a novel without letting AI write the book.)
Pressure-testing a scene. "What would a skeptical reader find unconvincing here?" You want honest friction.
Word-level help. "Give me ten verbs for how a tired dog walks." Small, fast, low risk.

Jobs where you should be careful

Drafting whole scenes in your voice. This is the job most likely to slowly replace your style with the model's. Some writers use it for a throwaway first pass and then rewrite every line. Others never do it. That choice is yours, but test for it deliberately so you know what the model does to your voice.
Line editing your prose. A model that "improves" every sentence often sands off what made it yours. The companion How to Use AI to Edit Your Own Draft Without Losing Your Voice goes deep on this.
Research facts inside fiction. Historical details, medical details, police procedure, and legal procedure all get invented confidently. Fiction gives you license to make things up, but readers notice when the made-up part is the thing you meant to get right. Check facts against real sources, and for anything about medicine, law, or weapons, talk to a qualified person or a solid reference rather than treating the model as the expert. AI explains; it doesn't decide.

Pick two or three jobs from these lists. Those are your test categories.

The free ways to run a model locally

You have a few free, well-documented ways to run a model on your own machine:

Ollama. Free, open-source, runs on macOS, Windows, and Linux. You pick models from its online library and run them with a short command such as ollama run followed by a model name. The official command reference lists ollama pull to download, ollama ls to list what you have, and ollama rm to remove one (Ollama CLI docs).
LM Studio. A desktop app with a point-and-click interface for downloading and chatting with local models. Apiary compares the two in LM Studio vs Ollama for Writers.
Open WebUI on top of Ollama. A browser-style chat window for Ollama. Setup is covered in Open WebUI + Ollama Setup for Beginners.

Ollama's own FAQ says that when you run models locally, Ollama does not see your prompts or data (Ollama FAQ). The same FAQ notes that Ollama also offers cloud-hosted models and explains how to switch on a local-only mode that turns cloud features off. For an unpublished manuscript, that local-only setting is worth knowing about. Check the FAQ for the current setting name before you rely on it.

Picking your candidates

You are going to test three to five models. More than that and you will lose track. Fewer than three and you have nothing to compare.

Rule 1: Fit before fame

A model that does not fit comfortably in your computer's memory will either refuse to load, crawl at a word every few seconds, or make the whole machine sluggish. A slow model is a model you will stop using. Start with what your machine can hold, not with the biggest name you have heard of.

The model library pages show download sizes. As a rough rule, the file has to fit in memory with room left over for your operating system, your writing app, and the model's working space for your text. I am not going to print a RAM table here, because the honest answer depends on the model, the quantization, and the length of text you feed it.

For background on why the same model comes in different sizes, read What Is a Quantized Model (Q4, Q8) in Plain English and the companion What Is a GGUF File and Why Local AI Uses It.

Rule 2: Mix sizes and families

Pick candidates from at least two different model families (different makers), and at least two sizes. Families have personalities. Some lean cautious and polite. Some are more willing to write dark or morally messy material, which a crime novelist may need. Size often affects how well a model holds a long chapter in its head, but bigger is not automatically better at voice.

Rule 3: Prefer "instruct" or "chat" versions for this test

Most model families come in a base version and an instruction-tuned version. The instruction-tuned one (often labeled "instruct" or "chat") is built to follow requests like "give me ten options." Base models are more like raw autocomplete. They can be interesting for experimental writing, but they are harder to steer. For a first bake-off, stick to instruct versions.

Rule 4: Read the model card before you fall in love

Every serious model comes with a model card: a page describing what it was built for, what it was trained on (sometimes), its known limits, and its license. Some licenses restrict commercial use or certain kinds of content. If you plan to publish and sell your book, read the license. You are not asking the model to own your words, but you should know the terms of the tool you used. Apiary's companion What Is a Model Card and How to Read One walks through this in plain English.

Build your fiction test kit

This is the heart of the method. You will give every candidate the exact same tasks, with the exact same wording, and save the answers side by side.

Make a folder called model-bakeoff. Inside it, create one text file per task. Here are six tasks that cover most fiction jobs. Adjust the details to your genre.

Task 1: The brainstorm test

I'm writing a quiet small-town mystery. The town librarian has been returning books to the shelf that nobody checked out. Give me 12 possible explanations, ranging from mundane to strange. One line each. No explanations of your reasoning.

What you are looking for: variety, specificity, and at least two ideas you would not have thought of. Also watch whether it obeys "one line each" and "no explanations." A model that cannot follow a simple format will fight you all the time.

Task 2: The voice test

Write a short paragraph (100 to 150 words) of your own prose. A real one from your work in progress, or a scene you write just for this test. Then:

Here is a paragraph in my voice. Continue for one more paragraph of about the same length. Match the sentence rhythm, the narrator's attitude, and the vocabulary level. Do not add any new characters.

What you are looking for: does the continuation sound like you, or like a generic narrator? Count how many words it uses that you would never use. Watch for a sudden jump into fancy vocabulary, sentimental endings, or a tidy moral.

Task 3: The dialogue test

Two brothers who haven't spoken in five years meet at their mother's hospital bed. Write 12 lines of dialogue only, no narration. One brother is trying to be polite. The other is angry but won't say why. Neither should say directly what they feel.

What you are looking for: subtext. Weak models make characters announce their feelings ("I'm angry because you left!"). Better ones let tension sit under the words. There is a whole companion on writing dialogue with AI help without flat characters.

Task 4: The critique test

Paste a scene you know has a problem. Then:

Read this scene as a skeptical editor. List the three biggest weaknesses, quoting the exact sentence for each. Do not rewrite anything. Do not praise anything.

What you are looking for: honesty and precision. Many models default to compliments ("This is a compelling scene!") even when told not to. A useful critique partner quotes the actual weak lines. If it quotes sentences that are not in your scene, that is a hallucination. Score it down hard.

Task 5: The continuity test

Take two passages from your manuscript where you know a detail appears (an eye color, a street name, the day of the week). Change the detail in the second passage on purpose. Then:

Compare these two passages. List any facts about characters, places, or timeline that contradict each other. Quote both versions.

What you are looking for: does it catch your planted error? Does it invent errors that are not there? This test matters for long projects.

Task 6: The dark-material test (only if your genre needs it)

Crime, horror, war fiction, and literary fiction often need violence, grief, cruelty, or moral ugliness on the page. Some models refuse or soften these scenes. Write a request that fits your genre and that you would be comfortable publishing. For example:

In my crime novel, a detective finds the aftermath of a burglary that turned violent. Describe the room in 150 words from the detective's tired point of view. Focus on small physical details, not gore.

What you are looking for: does the model engage with the scene at a level that fits your book, or does it lecture you, refuse, or water it down into a greeting card? This is not about pushing a model to produce harmful instructions. It is about whether it can handle the normal emotional range of adult fiction.

Run the bake-off

Keep the settings the same

Most local tools let you change "temperature," a setting that controls how adventurous the word choices are. Higher tends to be more varied and more chaotic; lower tends to be more predictable. Ollama's Modelfile reference describes temperature this way and lists a default value (Ollama Modelfile reference). For a fair test, leave every model at its default the first time. Change one thing at a time. See What Is Model Temperature in Plain English if you want to understand the dial.

Start a fresh chat for every task

Do not run all six tasks in one long conversation. Earlier answers pollute later ones. Start a new chat for each task with each model. In Ollama's terminal chat, you can quit and run the model again. In app interfaces, use the "new chat" button.

Run each task twice

Models are a little random. One great answer can be luck. Run each task twice per model and judge the pair. If one run is brilliant and the other is mush, the model is inconsistent, and that is worth knowing.

Save everything

Copy each answer into a file named like task2-voice_modelA_run1.txt. This feels tedious. It is the difference between a real decision and a vague impression. When you come back in a week, you will be glad you can reread them.

Time it, roughly

Note how long each model takes to start answering and finish. You do not need a stopwatch. "Felt instant," "a few seconds," and "went to make coffee" are enough. A model that makes you wait for every brainstorm will break your creative flow.

Score it with a simple sheet

Make a table. Rows are models. Columns are your test tasks plus speed. Score each cell 1 to 5.

Model	Brainstorm	Voice	Dialogue	Critique	Continuity	Speed	Notes
Candidate A
Candidate B
Candidate C

A few scoring rules keep you honest:

Read blind if you can. Ask a friend to paste the outputs into one document with the model names removed and labeled X, Y, Z. You will be surprised how often your favorite name loses.
Weight the columns that match your jobs. If you only want brainstorming and continuity help, those columns count double. Voice imitation may not matter to you at all.
Penalize disobedience. If you said "no narration" and it narrated, take a point off. Format-following predicts how much you will fight the tool later.
Penalize invented quotes. In the critique and continuity tasks, any quote that is not actually in your text is a serious problem. A model that makes up evidence about your own manuscript cannot be trusted as a checker.
Write one sentence of notes per model. "Great ideas, terrible at following format." "Sounds like a greeting card." "Caught my planted error both times." These notes will decide ties.

What good and bad look like

You will start to recognize patterns quickly. Here are the ones fiction writers mention most often.

Signs of a model that flattens fiction

Every character speaks in complete, grammatical, emotionally explicit sentences.
Scenes end with a little lesson or a hopeful turn you did not ask for.
The same handful of words keep showing up across different tasks, often fancy abstract ones. Apiary's companion on finding repetitive words in your draft shows how to catch these.
Villains explain themselves at length.
It praises your writing before critiquing, even when told not to.
Its "continuation" of your paragraph turns your plain narrator into a lyrical one.

Signs of a model that can help

It follows the format you asked for, the first time.
Brainstorm lists include specific, concrete, slightly odd ideas, not just generic ones.
Dialogue has people avoiding things, interrupting, and talking past each other.
Critique quotes your real sentences and points at real problems.
It says "I don't see a contradiction" when there isn't one, instead of inventing one to seem useful.

No local model will hit every point. You are looking for the one whose weaknesses you can live with for the jobs you actually do.

Tuning after you choose

Once you have a winner, a few free adjustments can make it a better partner.

Write a standing instruction

Most tools let you set a "system prompt": standing instructions the model sees before every chat. For fiction, keep it short and specific:

You are a brainstorming partner for a novelist. Give options, not finished prose, unless asked. Keep lists to one line per item. When critiquing, quote the exact sentence and never praise. Never invent quotes from the manuscript.

Apiary has a full guide on what a system prompt is and how to write one for your local model. If you use Ollama, you can save these instructions permanently into your own named version of a model with a Modelfile. The companion Ollama Modelfiles in Plain English shows how.

Keep a style sheet the model can read

For long projects, keep a one-page file with character names, ages, eye colors, places, and rules of your world. Paste it in when you ask continuity questions. The companion How to Keep a Writing Style Guide Your Local AI Can Follow covers the format.

Watch the context window

Every model has a limit on how much text it can "see" at once, called the context window. If you paste three chapters and ask about the first page, the model may have effectively lost track of the beginning. Ollama's docs describe a num_ctx setting for this and explain that larger windows use more memory (Ollama Modelfile reference; Ollama FAQ). For continuity checks, feed shorter passages rather than whole books. Background reading: What Is a Context Window in Plain English.

Consider a second model for a second job

It is fine to keep two models: a fast small one for brainstorming and word help, and a slower larger one for reading long passages. Ollama lets you see what you have installed with ollama ls and remove what you don't use with ollama rm (Ollama CLI docs). Apiary's How to Run Two Local Models and Compare Their Answers covers doing this deliberately.

Protecting your voice while you use it

The biggest risk in fiction is not a bad model. It is a decent model you use so often that your book slowly starts sounding like it. A few habits help.

Ask for options, then choose

When the model gives you ten options, you are still the author. When it gives you one finished paragraph, it is tempting to accept it. Default to lists.

Retype instead of pasting

If you like an idea from the model, write your own sentence for it instead of pasting the model's sentence. This one habit keeps your voice in charge more than any setting.

Keep a "never" list

Note the words and moves your model loves that you hate. When you see them in your draft, you know where they came from.

Do a monthly read-aloud

Once a month, read a recent chapter out loud. If parts sound like a stranger, check whether those were the parts where you leaned on the model. Apiary's How to Proof Your Draft by Ear With Free Text-to-Speech shows how to use your computer's built-in voice for this.

Privacy for unpublished work

A running local model is one of the more private ways to get AI help with a manuscript, because the text does not need to leave your computer. Three cautions still apply:

Check that you are actually local. Some apps offer both local and cloud models in the same menu. Make sure the one you pick is the one on your machine. Ollama's FAQ describes how to disable its cloud features if you want to be sure.
Back up your chats and notes deliberately. If your brainstorming lives only inside a chat app, a crashed laptop can take it with it. See How to Back Up Your Local AI Models and Chats Before a Laptop Dies.
Be careful with "helpful" plugins. Browser extensions and add-ons that connect to your local model can also connect to other places. Only install ones you understand.

More on keeping things private: Private ChatGPT Alternative: Local AI That Stays on Your Machine.

When to repeat the bake-off

Models improve, and new ones appear in the libraries constantly. You do not need to chase every release. Repeat your bake-off when:

You start a new project in a different genre.
Your current model keeps annoying you in the same way.
You get a new computer that can hold larger models.
A model you use gets updated and starts behaving differently.

Because you saved your test kit, re-running it takes an hour, not an evening. Keep your old scores, too. They show you whether "new" actually means "better for you." Before you update a model you rely on, read How to Update Local AI Models Without Breaking Your Setup so you can roll back if the new version writes worse.

A sample evening plan

Here is how the whole thing looks when you sit down to do it.

Hour one: setup. Install Ollama or LM Studio if you have not already. Write your six test files. Pick three candidate models that fit your machine, ideally from two families and two sizes. Start the downloads; they can take a while on slow internet.

Hour two: run the tests. For each model, run each task twice in fresh chats. Save each answer to a file. Note rough speed.

Hour three: score. Make the score sheet. Read outputs, blind if you can. Score 1 to 5. Write one sentence of notes per model. Pick a winner and a runner-up.

Last fifteen minutes: set up the winner. Write a short system prompt for it. Save it, or make a Modelfile. Delete the models you know you will never use so they stop taking up disk space. Put your test kit somewhere you will find it next year.

Common questions

Is a local model good enough for fiction?

For brainstorming, continuity checks, reverse outlines, and critique, many free local models are useful. For drafting finished prose in your voice, results vary a lot, and many writers prefer not to use any model for that job. Test it on your own work rather than trusting anyone's general claim, including mine.

Do I need a powerful computer?

You need enough memory to hold the model comfortably. Smaller models run on ordinary laptops, more slowly on older ones. Larger models need more memory and benefit from a graphics card or recent Apple chip. Check the download size on the model's library page and your computer's memory before you choose.

Will the model steal my story?

A model running locally on your computer, with cloud features off, is not sending your manuscript anywhere by design. Cloud chat services are different, and their terms vary. Read the privacy terms of any cloud tool before pasting unpublished work into it.

Can I publish a book I wrote with AI help?

Many writers do, and some publishers and contests have rules about AI use. Rules and laws are changing. Check the guidelines of any publisher, agent, or contest you submit to, and if you have real money or rights at stake, talk to an intellectual-property attorney. AI can explain the general landscape, but it doesn't decide your legal position.

Should I fine-tune a model on my own writing?

Probably not as a first step. Good instructions and a style sheet get most writers most of the way, for much less effort. Apiary's Should You Fine-Tune a Local Model or Just Prompt Better explains the tradeoff.

The short version

Decide which fiction jobs you want help with.
Pick three to five free models that fit your machine, from at least two families.
Read each model's card and license.
Give every model the same six tasks, in fresh chats, twice each.
Score them on a simple sheet, blind if you can, and penalize invented quotes.
Keep the winner, give it standing instructions, and delete the rest.
Protect your voice: ask for options, retype what you keep, read aloud monthly.

The right model is not the one with the biggest number next to its name. It is the one that makes you want to keep writing your own book.

Sources checked

Ollama CLI reference: https://docs.ollama.com/cli (checked 2026-10-04)
Ollama FAQ (local data handling, cloud disable, context length, model storage): https://docs.ollama.com/faq (checked 2026-10-04)
Ollama Modelfile reference (temperature, num_ctx, SYSTEM): https://docs.ollama.com/modelfile (checked 2026-10-04)
Hugging Face, Model Cards documentation: https://huggingface.co/docs/hub/model-cards (checked 2026-10-04)