There Are No Rules
"One night [at university] a friend lent me a book of short stories by Franz Kafka. I went back to the pension where I was staying and began to read The Metamorphosis. The first line almost knocked me off the bed, I was so surprised. The first line reads, 'As Gregor Samsa awoke that morning from uneasy dreams, he found himself transformed in his bed into a gigantic insect...' When I read the line I thought to myself that I didn't know that anyone was allowed to write things like that. If I had known, I would have started writing a long time ago."
- Gabriel Garcia Marquez
As quoted in The Modern Library Writer's Workshop by Stephen Koch.
Art transcends rules. As soon as someone says, "Don't," some writer will come along and do exactly what was forbidden, and make it work. I think the quote above captures the danger of a rule-based approach to art: it leads young writers to the belief that some things are not allowed. As if art is answerable to some cosmic hall-monitor, telling you where you can and cannot go.
All things are allowed.
At the deepest level, a writer has to follow their muse, find their own voice, take their writing where it demands, not where some external (or internal) monitor says it should go.
This does not mean that choices don't have consequences. One needs to know the effect one is looking to create and what choices will achieve that effect, as well as what choices will undermine that effect. Technique is not a matter of learning rules, but of familiarizing oneself with the full range of effects that have been achieved in good writing, understanding the connection between one's choices and the effect of those choices, and developing skills in executing the broad range of choices available.
For instance, one of the effects that much fiction strives for is to create an ongoing fictional dream for the reader, in which he or she stops seeing words on a page, and is pulled into the experience of the fictional scene as it unfolds--seeing the rescue plane take off before the hero reaches it; hearing the dog (the reader hopes it's only a dog) rustling through the garbage in the alley below the bedroom window; feeling the hot sand of the Cote d'Azur beneath their feet. And many of the "rules" in fiction derive from the fact that poor grammar, unclear sentences, confusion about who is speaking, phony-sounding dialog, large dumps of exposition, use of cliches, lack of clarity with POV, etc. can all pull the reader out of that fictional dream, and instead leave them confused or annoyed, or even angry, over the writing. (Of course, striving for that fictional dream isn't a rule. Some fiction works by purposefully breaking the fictional dream. John Barth's short story "Lost in the Funhouse" tells a story about a boy and his family going to an amusement park, but intentionally interrupts that narrative to point out the techniques used in telling the story.)
The fact there are no absolute rules also does not mean that writers will never choose to work within a framework: adopting the rules of a given form, the expectations of a specific genre, or the requirements of a particular market.
For example, much of the best poetry in English--from Chaucer to Shakespeare to Keats to Robert Frost--is written in iambic pentameter. A great many English poems make use of a formal rhyme scheme. Both meter and rhyme are somewhat out of fashion these days, but they certainly remain viable options to a contemporary poet. The villanelle is one of the most restrictive poetic forms--specifying not just a rhyme scheme but a specific pattern for repeating whole lines. Yet can you imagine the beauty of Dylan Thomas's Do Not Go Gentle Into that Good Night not expressed through the villanelle form?
In terms of genre, readers will have certain genre expectations. If you're writing mysteries, readers are going to expect a crime, or some mystery, to be solved. You can of course break that rule, but at a certain point, it might become meaningless to call your story a mystery. If you are writing a literary short story, opening with a star ship captain chasing down aliens with his ray gun is going to turn off many readers who have different expectations of the literary genre.
And of course given markets have guidelines. If one is writing for Analog, there needs to be a speculative element in the story. If one is writing a screenplay one hopes to sell to a major studio, one's chances greatly go up if it is at least one and a half hours long, but not (Return of the King notwithstanding) three and a half hours long.
So the writer will always have to deal with the effect their choices achieve--intended or not. And a given writer might choose to adopt the rules of a given form, genre or market. That said, at the deepest level, there are no rules.