How can i use the values from hashset (the docid and offset) to the reduce writable so as to connect map writable with reduce writable?
The mapper (LineIndexMapper) works fine but in the reducer (LineIndexReducer) i get the error that it can't get string as argument when i type this:
context.write(key, new IndexRecordWritable("some string");
although i have the public String toString() in the ReduceWritable too.
I believe the hashset in reducer's writable (IndexRecordWritable.java) maybe isn't taking the values correctly?
I have the below code.
IndexMapRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
public class IndexMapRecordWritable implements Writable {
private LongWritable offset;
private Text docid;
public LongWritable getOffsetWritable() {
return offset;
}
public Text getDocidWritable() {
return docid;
}
public long getOffset() {
return offset.get();
}
public String getDocid() {
return docid.toString();
}
public IndexMapRecordWritable() {
this.offset = new LongWritable();
this.docid = new Text();
}
public IndexMapRecordWritable(long offset, String docid) {
this.offset = new LongWritable(offset);
this.docid = new Text(docid);
}
public IndexMapRecordWritable(IndexMapRecordWritable indexMapRecordWritable) {
this.offset = indexMapRecordWritable.getOffsetWritable();
this.docid = indexMapRecordWritable.getDocidWritable();
}
#Override
public String toString() {
StringBuilder output = new StringBuilder()
output.append(docid);
output.append(offset);
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
IndexRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.HashSet;
import org.apache.hadoop.io.Writable;
public class IndexRecordWritable implements Writable {
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
public IndexRecordWritable() {
}
public IndexRecordWritable(
Iterable<IndexMapRecordWritable> indexMapRecordWritables) {
}
#Override
public String toString() {
StringBuilder output = new StringBuilder();
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
Alright, here is my answer based on a few assumptions. The final output is a text file containing the key and the file names separated by a comma based on the information in the reducer class's comments on the pre-condition and post-condition.
In this case, you really don't need IndexRecordWritable class. You can simply write to your context using
context.write(key, new Text(valueBuilder.substring(0, valueBuilder.length() - 1)));
with the class declaration line as
public class LineIndexReducer extends Reducer<Text, IndexMapRecordWritable, Text, Text>
Don't forget to set the correct output class in the driver.
That must serve the purpose according to the post-condition in your reducer class. But, if you really want to write a Text-IndexRecordWritable pair to your context, there are two ways approach it -
with string as an argument (based on your attempt passing a string when you IndexRecordWritable class constructor is not designed to accept strings) and
with HashSet as an argument (based on the HashSet initialised in IndexRecordWritable class).
Since your constructor of IndexRecordWritable class is not designed to accept String as an input, you cannot pass a string. Hence the error you are getting that you can't use string as an argument. Ps: if you want your constructor to accept Strings, you must have another constructor in your IndexRecordWritable class as below:
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
}
// to accpet string
public IndexRecordWritable (String value) {
this.value = value;
}
but that won't be valid if you want to use the HashSet. So, approach #1 can't be used. You can't pass a string.
That leaves us with approach #2. Passing a HashSet as an argument since you want to make use of the HashSet. In this case, you must create a HashSet in your reducer before passing it as an argument to IndexRecordWritable in context.write.
To do this, your reducer must look like this.
#Override
protected void reduce(Text key, Iterable<IndexMapRecordWritable> values, Context context) throws IOException, InterruptedException {
//StringBuilder valueBuilder = new StringBuilder();
HashSet<IndexMapRecordWritable> set = new HashSet<>();
for (IndexMapRecordWritable val : values) {
set.add(val);
//valueBuilder.append(val);
//valueBuilder.append(",");
}
//write the key and the adjusted value (removing the last comma)
//context.write(key, new IndexRecordWritable(valueBuilder.substring(0, valueBuilder.length() - 1)));
context.write(key, new IndexRecordWritable(set));
//valueBuilder.setLength(0);
}
and your IndexRecordWritable.java must have this.
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
//private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
tokens.addAll(indexMapRecordWritables);
}
Remember, this is not the requirement according to the description of your reducer where it says.
POST-CONDITION: emit the output a single key-value where all the file names are separated by a comma ",". <"marcello", "a.txt#3345,b.txt#344,c.txt#785">
If you still choose to emit (Text, IndexRecordWritable), remember to process the HashSet in IndexRecordWritable to get it in the desired format.
I want to import the following file with Spring Batch
key;value
A;9,5
I model it with the bean
class CsvModel
{
String key
Double value
}
The shown code here is Groovy but the language is irrelevant for the problem.
#Bean
#StepScope
FlatFileItemReader<CsvModel> reader2()
{
// set the locale for the tokenizer, but this doesn't solve the problem
def locale = Locale.getDefault()
def fieldSetFactory = new DefaultFieldSetFactory()
fieldSetFactory.setNumberFormat(NumberFormat.getInstance(locale))
def tokenizer = new DelimitedLineTokenizer(';')
tokenizer.setNames([ 'key', 'value' ].toArray() as String[])
// and assign the fieldSetFactory to the tokenizer
tokenizer.setFieldSetFactory(fieldSetFactory)
def fieldMapper = new BeanWrapperFieldSetMapper<CsvModel>()
fieldMapper.setTargetType(CsvModel.class)
def lineMapper = new DefaultLineMapper<CsvModel>()
lineMapper.setLineTokenizer(tokenizer)
lineMapper.setFieldSetMapper(fieldMapper)
def reader = new FlatFileItemReader<CsvModel>()
reader.setResource(new FileSystemResource('output/export.csv'))
reader.setLinesToSkip(1)
reader.setLineMapper(lineMapper)
return reader
}
Setting up a reader is well known, what was new for me was the first code block, setting up a numberFormat / locale / fieldSetFactory and assign it to the tokenizer. However this doesn't work, I still receive the exception
Field error in object 'target' on field 'value': rejected value [5,0]; codes [typeMismatch.target.value,typeMismatch.value,typeMismatch.float,typeMismatch]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [target.value,value]; arguments []; default message [value]]; default message [Failed to convert property value of type 'java.lang.String' to required type 'float' for property 'value'; nested exception is java.lang.NumberFormatException: For input string: "9,5"]
at org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper.mapFieldSet(BeanWrapperFieldSetMapper.java:200) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.item.file.mapping.DefaultLineMapper.mapLine(DefaultLineMapper.java:43) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
at org.springframework.batch.item.file.FlatFileItemReader.doRead(FlatFileItemReader.java:180) ~[spring-batch-infrastructure-4.1.2.RELEASE.jar:4.1.2.RELEASE]
So the question is: how do I import floats in the locale de_AT (we write our decimals with a comma like this: 3,141592)? I could avoid this problem with a FieldSetMapper but I want to understand what's going on here and want to avoid the unnecessary mapper class.
And even the FieldSetMapper solution doesn't obey locales out of the box, I have to read a string and convert it myself in a double:
class PnwExportFieldSetMapper implements FieldSetMapper<CsvModel>
{
private nf = NumberFormat.getInstance(Locale.getDefault())
#Override
CsvModel mapFieldSet(FieldSet fieldSet) throws BindException
{
def model = new CsvModel()
model.key = fieldSet.readString(0)
model.value = nf.parse(fieldSet.readString(1)).doubleValue()
return model
}
}
The class DefaultFieldSet has a function setNumberFormat, but when and where do I call this function?
This unfortunately seems to be a bug. I have the same Problem and debugged into the code.
The BeanWrapperFieldSetMapper is not using the methods of DefaultFieldSetFactory, that would do the right conversion, but instead just uses FieldSet.getProperties and does the conversion by itself.
So, I see the following options: Provide the BeanWrapperFieldSetMapper either with PropertyEditors or a ConversionService, or use a different mapper.
Here is a sketch of a conversion Service:
private static class CS implements ConversionService {
#Override
public boolean canConvert(Class<?> sourceType, Class<?> targetType) {
return sourceType == String.class && targetType == double.class;
}
#Override
public boolean canConvert(TypeDescriptor sourceType, TypeDescriptor targetType) {
return sourceType.equals(TypeDescriptor.valueOf(String.class)) &&
targetType.equals(TypeDescriptor.valueOf(double.class)) ;
}
#Override
public <T> T convert(Object source, Class<T> targetType) {
return (T)Double.valueOf(source.toString().replace(',', '.'));
}
#Override
public Object convert(Object source, TypeDescriptor sourceType, TypeDescriptor targetType) {
return Double.valueOf(source.toString().replace(',', '.'));
}
}
and use it:
final BeanWrapperFieldSetMapper<IBISRecord> mapper = new BeanWrapperFieldSetMapper<>();
mapper.setTargetType(YourClass.class);
mapper.setConversionService(new CS());
...
new FlatFileItemReaderBuilder<IBISRecord>()
.name("YourReader")
.delimited()
.delimiter(";")
.includedFields(fields)
.names(names)
.fieldSetMapper(mapper)
.saveState(false)
.resource(resource)
.build();
My Problem is, that if I save a Document with a $ inside the content, Mongodb gives me an exception:
java.lang.IllegalArgumentException: Invalid BSON field name $ xxx
I would like that mongodb ignores the $ character in the content.
My Application is written in java. I read the content of the File and put it as a string into an object. After that the object will be saved with a MongoRepository class.
Someone has any ideas??
Example content
Edit: I heard mongodb has the same problem wit dot. Our Springboot found i workaround with dot, but not for dollar.
How to configure mongo converter in spring to encode all dots in the keys of map being saved in mongo db
If you are using Spring Boot you can extend MappingMongoConverter class and add override methods that do the escaping/unescaping.
#Component
public class MappingMongoConverterCustom extends MappingMongoConverter {
protected #Nullable
String mapKeyDollarReplacemant = "characters_to_replace_dollar";
protected #Nullable
String mapKeyDotReplacement = "characters_to_replace_dot";
public MappingMongoConverterCustom(DbRefResolver dbRefResolver, MappingContext<? extends MongoPersistentEntity<?>, MongoPersistentProperty> mappingContext) {
super(dbRefResolver, mappingContext);
}
#Override
protected String potentiallyEscapeMapKey(String source) {
if (!source.contains(".") && !source.contains("$")) {
return source;
}
if (mapKeyDotReplacement == null && mapKeyDollarReplacemant == null) {
throw new MappingException(String.format(
"Map key %s contains dots or dollars but no replacement was configured! Make "
+ "sure map keys don't contain dots or dollars in the first place or configure an appropriate replacement!",
source));
}
String result = source;
if(result.contains(".")) {
result = result.replaceAll("\\.", mapKeyDotReplacement);
}
if(result.contains("$")) {
result = result.replaceAll("\\$", mapKeyDollarReplacemant);
}
//add any other replacements you need
return result;
}
#Override
protected String potentiallyUnescapeMapKey(String source) {
String result = source;
if(mapKeyDotReplacement != null) {
result = result.replaceAll(mapKeyDotReplacement, "\\.");
}
if(mapKeyDollarReplacemant != null) {
result = result.replaceAll(mapKeyDollarReplacemant, "\\$");
}
//add any other replacements you need
return result;
}
}
If you go with this approach make sure you override the default converter from AbstractMongoConfiguration like below:
#Configuration
public class MongoConfig extends AbstractMongoConfiguration{
#Bean
public DbRefResolver getDbRefResolver() {
return new DefaultDbRefResolver(mongoDbFactory());
}
#Bean
#Override
public MappingMongoConverter mappingMongoConverter() throws Exception {
MappingMongoConverterCustom converter = new MappingMongoConverterCustom(getDbRefResolver(), mongoMappingContext());
converter.setCustomConversions(customConversions());
return converter;
}
.... whatever you might need extra ...
}
The code looks like this:
StringBuilder builder = new StringBuilder();
XmlWriterSettings settings = new XmlWriterSettings();
settings.OmitXmlDeclaration = true;
using (XmlWriter xmlWriter = XmlWriter.Create(builder, settings))
{
XmlSerializer s = new XmlSerializer(objectToSerialize.GetType());
s.Serialize(xmlWriter, objectToSerialize);
}
The resulting serialized document includes namespaces, like so:
<message xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"
xmlns="urn:something">
...
</message>
To remove the xsi and xsd namespaces, I can follow the answer from How to serialize an object to XML without getting xmlns=”…”?.
I want my message tag as <message> (without any namespace attributes). How can I do this?
...
XmlSerializer s = new XmlSerializer(objectToSerialize.GetType());
XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
ns.Add("","");
s.Serialize(xmlWriter, objectToSerialize, ns);
This is the 2nd of two answers.
If you want to just strip all namespaces arbitrarily from a document during serialization, you can do this by implementing your own XmlWriter.
The easiest way is to derive from XmlTextWriter and override the StartElement method that emits namespaces. The StartElement method is invoked by the XmlSerializer when emitting any elements, including the root. By overriding the namespace for each element, and replacing it with the empty string, you've stripped the namespaces from the output.
public class NoNamespaceXmlWriter : XmlTextWriter
{
//Provide as many contructors as you need
public NoNamespaceXmlWriter(System.IO.TextWriter output)
: base(output) { Formatting= System.Xml.Formatting.Indented;}
public override void WriteStartDocument () { }
public override void WriteStartElement(string prefix, string localName, string ns)
{
base.WriteStartElement("", localName, "");
}
}
Suppose this is the type:
// explicitly specify a namespace for this type,
// to be used during XML serialization.
[XmlRoot(Namespace="urn:Abracadabra")]
public class MyTypeWithNamespaces
{
// private fields backing the properties
private int _Epoch;
private string _Label;
// explicitly define a distinct namespace for this element
[XmlElement(Namespace="urn:Whoohoo")]
public string Label
{
set { _Label= value; }
get { return _Label; }
}
// this property will be implicitly serialized to XML using the
// member name for the element name, and inheriting the namespace from
// the type.
public int Epoch
{
set { _Epoch= value; }
get { return _Epoch; }
}
}
Here's how you would use such a thing during serialization:
var o2= new MyTypeWithNamespaces { ..intializers.. };
var builder = new System.Text.StringBuilder();
using ( XmlWriter writer = new NoNamespaceXmlWriter(new System.IO.StringWriter(builder)))
{
s2.Serialize(writer, o2, ns2);
}
Console.WriteLine("{0}",builder.ToString());
The XmlTextWriter is sort of broken, though. According to the reference doc, when it writes it does not check for the following:
Invalid characters in attribute and element names.
Unicode characters that do not fit the specified encoding. If the Unicode
characters do not fit the specified
encoding, the XmlTextWriter does not
escape the Unicode characters into
character entities.
Duplicate attributes.
Characters in the DOCTYPE public
identifier or system identifier.
These problems with XmlTextWriter have been around since v1.1 of the .NET Framework, and they will remain, for backward compatibility. If you have no concerns about those problems, then by all means use the XmlTextWriter. But most people would like a bit more reliability.
To get that, while still suppressing namespaces during serialization, instead of deriving from XmlTextWriter, define a concrete implementation of the abstract XmlWriter and its 24 methods.
An example is here:
public class XmlWriterWrapper : XmlWriter
{
protected XmlWriter writer;
public XmlWriterWrapper(XmlWriter baseWriter)
{
this.Writer = baseWriter;
}
public override void Close()
{
this.writer.Close();
}
protected override void Dispose(bool disposing)
{
((IDisposable) this.writer).Dispose();
}
public override void Flush()
{
this.writer.Flush();
}
public override string LookupPrefix(string ns)
{
return this.writer.LookupPrefix(ns);
}
public override void WriteBase64(byte[] buffer, int index, int count)
{
this.writer.WriteBase64(buffer, index, count);
}
public override void WriteCData(string text)
{
this.writer.WriteCData(text);
}
public override void WriteCharEntity(char ch)
{
this.writer.WriteCharEntity(ch);
}
public override void WriteChars(char[] buffer, int index, int count)
{
this.writer.WriteChars(buffer, index, count);
}
public override void WriteComment(string text)
{
this.writer.WriteComment(text);
}
public override void WriteDocType(string name, string pubid, string sysid, string subset)
{
this.writer.WriteDocType(name, pubid, sysid, subset);
}
public override void WriteEndAttribute()
{
this.writer.WriteEndAttribute();
}
public override void WriteEndDocument()
{
this.writer.WriteEndDocument();
}
public override void WriteEndElement()
{
this.writer.WriteEndElement();
}
public override void WriteEntityRef(string name)
{
this.writer.WriteEntityRef(name);
}
public override void WriteFullEndElement()
{
this.writer.WriteFullEndElement();
}
public override void WriteProcessingInstruction(string name, string text)
{
this.writer.WriteProcessingInstruction(name, text);
}
public override void WriteRaw(string data)
{
this.writer.WriteRaw(data);
}
public override void WriteRaw(char[] buffer, int index, int count)
{
this.writer.WriteRaw(buffer, index, count);
}
public override void WriteStartAttribute(string prefix, string localName, string ns)
{
this.writer.WriteStartAttribute(prefix, localName, ns);
}
public override void WriteStartDocument()
{
this.writer.WriteStartDocument();
}
public override void WriteStartDocument(bool standalone)
{
this.writer.WriteStartDocument(standalone);
}
public override void WriteStartElement(string prefix, string localName, string ns)
{
this.writer.WriteStartElement(prefix, localName, ns);
}
public override void WriteString(string text)
{
this.writer.WriteString(text);
}
public override void WriteSurrogateCharEntity(char lowChar, char highChar)
{
this.writer.WriteSurrogateCharEntity(lowChar, highChar);
}
public override void WriteValue(bool value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(DateTime value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(decimal value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(double value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(int value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(long value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(object value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(float value)
{
this.writer.WriteValue(value);
}
public override void WriteValue(string value)
{
this.writer.WriteValue(value);
}
public override void WriteWhitespace(string ws)
{
this.writer.WriteWhitespace(ws);
}
public override XmlWriterSettings Settings
{
get
{
return this.writer.Settings;
}
}
protected XmlWriter Writer
{
get
{
return this.writer;
}
set
{
this.writer = value;
}
}
public override System.Xml.WriteState WriteState
{
get
{
return this.writer.WriteState;
}
}
public override string XmlLang
{
get
{
return this.writer.XmlLang;
}
}
public override System.Xml.XmlSpace XmlSpace
{
get
{
return this.writer.XmlSpace;
}
}
}
Then, provide a derived class that overrides the StartElement method, as before:
public class NamespaceSupressingXmlWriter : XmlWriterWrapper
{
//Provide as many contructors as you need
public NamespaceSupressingXmlWriter(System.IO.TextWriter output)
: base(XmlWriter.Create(output)) { }
public NamespaceSupressingXmlWriter(XmlWriter output)
: base(XmlWriter.Create(output)) { }
public override void WriteStartElement(string prefix, string localName, string ns)
{
base.WriteStartElement("", localName, "");
}
}
And then use this writer like so:
var o2= new MyTypeWithNamespaces { ..intializers.. };
var builder = new System.Text.StringBuilder();
var settings = new XmlWriterSettings { OmitXmlDeclaration = true, Indent= true };
using ( XmlWriter innerWriter = XmlWriter.Create(builder, settings))
using ( XmlWriter writer = new NamespaceSupressingXmlWriter(innerWriter))
{
s2.Serialize(writer, o2, ns2);
}
Console.WriteLine("{0}",builder.ToString());
Credit for this to Oleg Tkachenko.
After reading Microsoft's documentation and several solutions online, I have discovered the solution to this problem. It works with both the built-in XmlSerializer and custom XML serialization via IXmlSerialiazble.
To wit, I'll use the same MyTypeWithNamespaces XML sample that's been used in the answers to this question so far.
[XmlRoot("MyTypeWithNamespaces", Namespace="urn:Abracadabra", IsNullable=false)]
public class MyTypeWithNamespaces
{
// As noted below, per Microsoft's documentation, if the class exposes a public
// member of type XmlSerializerNamespaces decorated with the
// XmlNamespacesDeclarationAttribute, then the XmlSerializer will utilize those
// namespaces during serialization.
public MyTypeWithNamespaces( )
{
this._namespaces = new XmlSerializerNamespaces(new XmlQualifiedName[] {
// Don't do this!! Microsoft's documentation explicitly says it's not supported.
// It doesn't throw any exceptions, but in my testing, it didn't always work.
// new XmlQualifiedName(string.Empty, string.Empty), // And don't do this:
// new XmlQualifiedName("", "")
// DO THIS:
new XmlQualifiedName(string.Empty, "urn:Abracadabra") // Default Namespace
// Add any other namespaces, with prefixes, here.
});
}
// If you have other constructors, make sure to call the default constructor.
public MyTypeWithNamespaces(string label, int epoch) : this( )
{
this._label = label;
this._epoch = epoch;
}
// An element with a declared namespace different than the namespace
// of the enclosing type.
[XmlElement(Namespace="urn:Whoohoo")]
public string Label
{
get { return this._label; }
set { this._label = value; }
}
private string _label;
// An element whose tag will be the same name as the property name.
// Also, this element will inherit the namespace of the enclosing type.
public int Epoch
{
get { return this._epoch; }
set { this._epoch = value; }
}
private int _epoch;
// Per Microsoft's documentation, you can add some public member that
// returns a XmlSerializerNamespaces object. They use a public field,
// but that's sloppy. So I'll use a private backed-field with a public
// getter property. Also, per the documentation, for this to work with
// the XmlSerializer, decorate it with the XmlNamespaceDeclarations
// attribute.
[XmlNamespaceDeclarations]
public XmlSerializerNamespaces Namespaces
{
get { return this._namespaces; }
}
private XmlSerializerNamespaces _namespaces;
}
That's all to this class. Now, some objected to having an XmlSerializerNamespaces object somewhere within their classes; but as you can see, I neatly tucked it away in the default constructor and exposed a public property to return the namespaces.
Now, when it comes time to serialize the class, you would use the following code:
MyTypeWithNamespaces myType = new MyTypeWithNamespaces("myLabel", 42);
/******
OK, I just figured I could do this to make the code shorter, so I commented out the
below and replaced it with what follows:
// You have to use this constructor in order for the root element to have the right namespaces.
// If you need to do custom serialization of inner objects, you can use a shortened constructor.
XmlSerializer xs = new XmlSerializer(typeof(MyTypeWithNamespaces), new XmlAttributeOverrides(),
new Type[]{}, new XmlRootAttribute("MyTypeWithNamespaces"), "urn:Abracadabra");
******/
XmlSerializer xs = new XmlSerializer(typeof(MyTypeWithNamespaces),
new XmlRootAttribute("MyTypeWithNamespaces") { Namespace="urn:Abracadabra" });
// I'll use a MemoryStream as my backing store.
MemoryStream ms = new MemoryStream();
// This is extra! If you want to change the settings for the XmlSerializer, you have to create
// a separate XmlWriterSettings object and use the XmlTextWriter.Create(...) factory method.
// So, in this case, I want to omit the XML declaration.
XmlWriterSettings xws = new XmlWriterSettings();
xws.OmitXmlDeclaration = true;
xws.Encoding = Encoding.UTF8; // This is probably the default
// You could use the XmlWriterSetting to set indenting and new line options, but the
// XmlTextWriter class has a much easier method to accomplish that.
// The factory method returns a XmlWriter, not a XmlTextWriter, so cast it.
XmlTextWriter xtw = (XmlTextWriter)XmlTextWriter.Create(ms, xws);
// Then we can set our indenting options (this is, of course, optional).
xtw.Formatting = Formatting.Indented;
// Now serialize our object.
xs.Serialize(xtw, myType, myType.Namespaces);
Once you have done this, you should get the following output:
<MyTypeWithNamespaces>
<Label xmlns="urn:Whoohoo">myLabel</Label>
<Epoch>42</Epoch>
</MyTypeWithNamespaces>
I have successfully used this method in a recent project with a deep hierachy of classes that are serialized to XML for web service calls. Microsoft's documentation is not very clear about what to do with the publicly accesible XmlSerializerNamespaces member once you've created it, and so many think it's useless. But by following their documentation and using it in the manner shown above, you can customize how the XmlSerializer generates XML for your classes without resorting to unsupported behavior or "rolling your own" serialization by implementing IXmlSerializable.
It is my hope that this answer will put to rest, once and for all, how to get rid of the standard xsi and xsd namespaces generated by the XmlSerializer.
UPDATE: I just want to make sure I answered the OP's question about removing all namespaces. My code above will work for this; let me show you how. Now, in the example above, you really can't get rid of all namespaces (because there are two namespaces in use). Somewhere in your XML document, you're going to need to have something like xmlns="urn:Abracadabra" xmlns:w="urn:Whoohoo. If the class in the example is part of a larger document, then somewhere above a namespace must be declared for either one of (or both) Abracadbra and Whoohoo. If not, then the element in one or both of the namespaces must be decorated with a prefix of some sort (you can't have two default namespaces, right?). So, for this example, Abracadabra is the defalt namespace. I could inside my MyTypeWithNamespaces class add a namespace prefix for the Whoohoo namespace like so:
public MyTypeWithNamespaces
{
this._namespaces = new XmlSerializerNamespaces(new XmlQualifiedName[] {
new XmlQualifiedName(string.Empty, "urn:Abracadabra"), // Default Namespace
new XmlQualifiedName("w", "urn:Whoohoo")
});
}
Now, in my class definition, I indicated that the <Label/> element is in the namespace "urn:Whoohoo", so I don't need to do anything further. When I now serialize the class using my above serialization code unchanged, this is the output:
<MyTypeWithNamespaces xmlns:w="urn:Whoohoo">
<w:Label>myLabel</w:Label>
<Epoch>42</Epoch>
</MyTypeWithNamespaces>
Because <Label> is in a different namespace from the rest of the document, it must, in someway, be "decorated" with a namespace. Notice that there are still no xsi and xsd namespaces.
XmlSerializer sr = new XmlSerializer(objectToSerialize.GetType());
TextWriter xmlWriter = new StreamWriter(filename);
XmlSerializerNamespaces namespaces = new XmlSerializerNamespaces();
namespaces.Add(string.Empty, string.Empty);
sr.Serialize(xmlWriter, objectToSerialize, namespaces);
This is the first of my two answers to the question.
If you want fine control over the namespaces - for example if you want to omit some of them but not others, or if you want to replace one namespace with another, you can do this using XmlAttributeOverrides.
Suppose you have this type definition:
// explicitly specify a namespace for this type,
// to be used during XML serialization.
[XmlRoot(Namespace="urn:Abracadabra")]
public class MyTypeWithNamespaces
{
// private fields backing the properties
private int _Epoch;
private string _Label;
// explicitly define a distinct namespace for this element
[XmlElement(Namespace="urn:Whoohoo")]
public string Label
{
set { _Label= value; }
get { return _Label; }
}
// this property will be implicitly serialized to XML using the
// member name for the element name, and inheriting the namespace from
// the type.
public int Epoch
{
set { _Epoch= value; }
get { return _Epoch; }
}
}
And this serialization pseudo-code:
var o2= new MyTypeWithNamespaces() { ..initializers...};
ns.Add( "", "urn:Abracadabra" );
XmlSerializer s2 = new XmlSerializer(typeof(MyTypeWithNamespaces));
s2.Serialize(System.Console.Out, o2, ns);
You would get something like this XML:
<MyTypeWithNamespaces xmlns="urn:Abracadabra">
<Label xmlns="urn:Whoohoo">Cimsswybclaeqjh</Label>
<Epoch>97</Epoch>
</MyTypeWithNamespaces>
Notice that there is a default namespace on the root element, and there is also a distinct namespace on the "Label" element. These namespaces were dictated by the attributes decorating the type, in the code above.
The Xml Serialization framework in .NET includes the possibility to explicitly override the attributes that decorate the actual code. You do this with the XmlAttributesOverrides class and friends. Suppose I have the same type, and I serialize it this way:
// instantiate the container for all attribute overrides
XmlAttributeOverrides xOver = new XmlAttributeOverrides();
// define a set of XML attributes to apply to the root element
XmlAttributes xAttrs1 = new XmlAttributes();
// define an XmlRoot element (as if [XmlRoot] had decorated the type)
// The namespace in the attribute override is the empty string.
XmlRootAttribute xRoot = new XmlRootAttribute() { Namespace = ""};
// add that XmlRoot element to the container of attributes
xAttrs1.XmlRoot= xRoot;
// add that bunch of attributes to the container holding all overrides
xOver.Add(typeof(MyTypeWithNamespaces), xAttrs1);
// create another set of XML Attributes
XmlAttributes xAttrs2 = new XmlAttributes();
// define an XmlElement attribute, for a type of "String", with no namespace
var xElt = new XmlElementAttribute(typeof(String)) { Namespace = ""};
// add that XmlElement attribute to the 2nd bunch of attributes
xAttrs2.XmlElements.Add(xElt);
// add that bunch of attributes to the container for the type, and
// specifically apply that bunch to the "Label" property on the type.
xOver.Add(typeof(MyTypeWithNamespaces), "Label", xAttrs2);
// instantiate a serializer with the overrides
XmlSerializer s3 = new XmlSerializer(typeof(MyTypeWithNamespaces), xOver);
// serialize
s3.Serialize(System.Console.Out, o2, ns2);
The result looks like this;
<MyTypeWithNamespaces>
<Label>Cimsswybclaeqjh</Label>
<Epoch>97</Epoch>
</MyTypeWithNamespaces>
You have stripped the namespaces.
A logical question is, can you strip all namespaces from arbitrary types during serialization, without going through the explicit overrides? The answer is YES, and how to do it is in my next response.