How to Remove CTRL-M characters From a File in UNIX and Linux

I wanted to transfer some files from windows to Unix using FileZilla, but the problem arises when these files are transferred (Ascii or Binary mode both) and opened using VI we get ^M characters, also known as CTRL-M characters. I searched about this but the solutions were to remove these ^M characters when files are transferred using utilities. Is there any way that these ^M characters do not appear in the first place? Well, my search continues but I will share the solution which worked for me for removing control M characters i.e. CTRL-M or ^M characters. There are several UNIX commands e.g. dos2unix which can be used to convert a Windows or DOS generated files to UNIX one. You can also usesed command (stream editor) to remove CTRL-M characters without opening the file, very useful if you are removing CTRL-M characters from a large file. Alternatively, you can use VI command to open the file and replace ^M characters with nothing.

Unix uses a single Line Feed (LF) character as the line break. Windows/DOS uses 2 characters: Carriage Return/Line Feed (CR/LF). The control M characters appear in a file when you transfer then from Windows to UNIX (see How linux works ). So the answer is no, you cannot eliminate the CR\LF (unless you write all your text in one line) but you can still remove those CTRL-M characters by using one of the two mentioned ways:

2 ways to remove Control-M characters from a file in UNIX

Here is a couple of ways to delete control-M characters from a file, in the first way we try to use a command which converts a DOS file to UNIX format and in second way we literally remove control-M characters using UNIX commands e.g. sed or vi

1st Way

$ dos2unix abc.txt

As the name suggest the dos2unix command convert a DOS or Windows generated file to UNIX file i.e. it replaces all CRLF to LF. See,The Linux Command Line: A Complete Introduction tolearning more about dos2unix command.

2nd Way

You may need to do this when you import a text file from MS-DOS or (Window 8, 8.1, 10 or any other version), and forget to transfer it in ASCII or text mode. Here are a couple ways to do it, you can choose the one you like.

The easiest way is probably to use the stream editor sed to remove the ^M characters as shown in the following example

$ sed -e "s/^M//" filename > newfilename

To enter ^M, type CTRL-V, then CTRL-M. I mean, hold down the CTRL key then press V and M in succession. The sed command is really helpful in case of removing ^M characters from large and huge files because you don't need to open the files into VI or any other editor.

Btw, here is a screenshot of a how Control-M character looks like inside a file in UNIX, I am sure you have seen it before:

Btw, you can also remove ^M characters by opening the file inVI editor and replacing them with nothing as shown in the following command:

$ vi filename Inside vi [in ESC mode] type: :%s/^M//g

To enter ^M, type CTRL-V, then CTRL-M i.e. hold down the CTRL key then press V and M in succession.

If you are an Emacs fan then don't get disappointed because you can also remove CTRL M characters in Emacs by following the below steps:

Go to the beginning of the document

Type: M-x replace-string RET C-q C-m RET RET

where "RET" means and C-q and C-m mean .

See Practical Guide to Linux Commands, Editors, and Shell Programming 3rd Edition by Mark G. Sobell to learn more about UNIX commands and editors.

In short, control-m characters will get appended to a file when a file is transferred from windows to UNIX machine. There are four ways it can be removed.

Using vi editor:

:%s/^M//g

Using col command:

$ cat filename | col -b > newfilename

Using sed command:

$ sed 's/^M//g' filename > newfilename

Using dos2unix command:

$ dos2unix filename newfilename

Now, let's see a complete example of removing control m characters from a file. You can see below we have a file with ^M characters , after using sed command those will be removed and verified again by using the cat command.

$ cat -v jsk hi^M jsk^M $ cat jsk | col -b > jsk.new $ cat -v jsk.new hi jsk $ sed 's/^M//g' jsk > jsk.new2 $ cat -v jsk.new2 hi jsk

Note: Remember how to type control M characters in UNIX, just hold the control key and then press v and m to get the control-m character.

Other UNIX command tutorials you may like

How to create complex directory structure using mkdir -p option ( example ) 10 Examples of lsof command in UNIX ( examples ) How to find IP address from hostname in UNIX and Linux? (commands) How to find total size of a directory in UNIX? (command) 10 example of xargs command in UNIX? (see here) 5 example of sort command in UNIX? (see here) 10 basic networking commands in UNIX? (see here) How to close the telnet terminal in UNIX and Windows? ( see here )

References

DOS vs Unix Line endings