Hoisting Another by Her Own Petard : THE JOURNALIST AND THE MURDERER <i> by Janet Malcolm (Alfred A. Knopf: $18.95; </i> 162 pp.)

Some years ago, Joan Didion ended ne of her most famous essays--a piece that all her admirers remember, and, fortunately, that few of her subjects subsequently took to heart--with this gaunt, defiant coda: “Writers,” she warned, “are always selling somebody out.”

At times, Janet Malcolm’s brilliant and discomfiting new book, “The Journalist and the Murderer,” is an attempt to graft flesh onto the bones of that remark. Her formulation is, if anything, even more stark than Didion’s. “Every journalist,” she writes, “who is not too stupid or too full of himself to notice what is going on knows that what he does is morally indefensible. He is a kind of confidence man, preying on people’s vanity, ignorance or loneliness, gaining their trust and betraying them without remorse. . . . Journalists justify their treachery in various ways according to their temperaments. The more pompous talk about freedom of speech and ‘the public’s right to know’; the least talented talk about Art; the seemliest murmur about earning a living.”

Hard pounding, this, as the Duke of Wellington remarked at Waterloo. There is harder to come. In Malcolm’s account, the relationship between journalist and subject is inevitably one of seduction and betrayal. All the rest is wishful thinking on the subject’s part, duplicity on the journalist’s.

The best thing for a wronged subject to do, Malcolm suggests, after what she calls his inevitable “dehoaxing,” is to consign the experience “to the rubbish heap of love affairs that ended badly.” The worst thing is to sue. But, of course, it is precisely the subject who can’t let go and the writer who denies his perfidy that engages Malcolm.

In 1984, Dr. Jeffrey MacDonald, a former Green Beret captain then serving three consecutive life sentences for the murders of his wife and two young children, sued Joe McGinniss, the author of a best-selling account of the case called “Fatal Vision.” MacDonald claimed that he had only cooperated with McGinniss--cooperation so unstinting that at times during his several trials he had treated McGinniss more like a member of his defense team than like a journalist--because the writer had asserted an unshakeable faith in his innocence.

The suit, MacDonald’s lawyer, Gary Bostwick, maintained, was “a case about a false friend.” For in between 1979, when he first approached MacDonald, and 1984, when the book appeared, McGinniss had become convinced that his subject was indeed a murderer. What made this change of heart bad faith was that after McGinniss had changed his mind, and was telling his publishers that “Fatal Vision” would portray the doctor as a killer, he continued to insist to MacDonald, even after the guilty verdict had come in, that he still thought the charges were false.

For Malcolm, the case of MacDonald v. McGinniss “was unprecedented in being concerned with a writer’s personal conduct toward his subject,” an opportunity to explore not only the moral ambiguity of journalism but what she regards as its essentially authoritarian nature. Approached by one of his lawyers, who wrote of the “grave threat to journalistic freedoms” that the case posed, Malcolm went to see McGinniss hoping, as she puts it rather too grandly, that their conversations would be like that of two experimenters “strolling home from the lab together after a day’s work, companionably thrashing out the problems of the profession.” Instead of finding a co-experimenter, however, Malcolm encountered another masochistic subject. McGinniss, she writes, with the brooding contempt that marked her portrait of him throughout the book, insisted that “we play the old game of Confession.”

Of course, the subject of “The Journalist and the Murderer” is writing, not crime. Nevertheless, Malcolm’s portrait of McGinniss is so damning and her portrait of MacDonald so noncommittal that one sometimes has to wonder about an approach in which a writer’s dishonesty is treated with more heat than the murder of three human beings.

Malcolm has always had a taste for demolition. As McGinniss’ defenders pointed out when the book first appeared as a series of articles in The New Yorker, when she talked about journalism as an ethically ambiguous activity she might have been thinking less of MacDonald v. McGinniss than of Masson v. Malcolm. A few years earlier, Malcolm herself had been sued by Jeffrey Masson, the former curator of the Freud archives whose character and competence she had effectively demolished in another series of brilliant New Yorker pieces. The piece on McGinniss, her critics insisted, was more an elaborate rationalization of what she had done to poor Masson than a truthful account either of McGinniss or of journalism.

Malcolm relies so heavily on psychotherapeutic, and usually rather rigidly Freudian, explanations for people’s behavior that she should not have been surprised when this rather dubious weapon was turned against her. In her afterword (there was no mention of Masson in the body of the text), Malcolm denies the connection. Readers will judge for themselves.

In the end, however, the question may be irrelevant to the book’s quality, if not to its author’s motivations. Like all of Malcolm’s best work, “The Journalist and the Murderer” is an attack, and what McGinniss and Masson really have in common is that they have been well and truly skewered by a writer as pitiless as she is talented.

Her masterpieces of severity will, as her subjects must know, endure long after their own work is forgotten. Indeed, the only recent piece of Malcolm’s that was a disappointment was her admiring portrait of Ingrid Sischy, the former editor of Art Forum magazine. She is not at her best when she is being kind, a fact she seems to acknowledge when she writes that “I, too, have committed the journalistic solecism of putting a person’s feelings above a text’s necessities.”

The limitation of the book is not that it is wrong but that, whatever she may claim, Malcolm’s subject is really not journalism at all. Malcolm affects to misunderstand the hostility with which her articles were received by working journalists, but the explanation is obvious. A reporter covering the START talks or Mandela’s release may have an ego as big as all outdoors, but his principal relation to his subject is neither of seducer nor of exploiter.

Moreover, Malcolm’s casting of the journalist in the position of authority and the subject in that of the victim ignores all those examples of the journalist as the tame and obedient creature of the powers that be. Think of the free ride the press gave John F. Kennedy, or Ronald Reagan during his first six years in office.

In fact, although Malcolm condescends to the notion of the journalist as artist, preposterously arguing that writing is best thought of as a way to make a living (why not go to law school then? The money’s better and the work exercises some of the same abilities), her jeremiad, at once so self-lacerating and so curiously complacent, applies not to working journalists, filing every day to deadline, but to writers like herself who produce long articles or books on subjects of their own choosing according to schedules, the money permitting, of their own devising, writers whose points of view are as important as their subject matter: in short, artists.

For it is not with regard to journalism but with regard to the making of works of art that Malcolm’s important book gathers its inspiration, its breathtaking rhetorical velocity, and its great truth.