I am working on a small ticketing prototype.
What I want to do is - if the top-level ADF is selected I want to return the AID of all the containing DFs and on first sight it works quite well.
I create the ADF, and 1 or 2 DFs. When the ADF is selected, the AIDs of those DFs are returned fine and I can add EFs (or DFs), ...
Now when I restart the whole thing (I am using JCOP btw.) I can still select the ADF but the AIDs from the DFs are not returned anymore, in fact I get a 6F00 "no precise diagnosis".
For my data structure - first you see the minimal constructor for the ADF, which has no parents
public DirectoryFile(byte[] aid) {
super(aid);
this.aid = aid;
numApp = 1;
created = true;
}
the second structure is the same but for a "usual" DirectoryFile with a parentDirectoryFile and an array (arrayFiles) of Elementary Files:
public DirectoryFile(byte[] aid, DirectoryFile parent) {
super(aid, parent);
for (byte i = 0; i < numberFiles; i++) {
arrayFiles[i].setActive(false);
}
}
both inherit from the same File.class
public File (byte aid[]) {
Util.arrayCopy(aid, (short) 0, this.aid, (short) 0, (short) 6);
}
public File (byte[] aid, DirectoryFile parentFile) {
this.parentFile = parentFile;
Util.arrayCopy(aid, (short) 0, this.aid, (short) 0, (short) 6);
}
This should be a very basic Filesystem and it does work as long as the card is connected to the terminal, but the information seems to be lost after restart of the program although I am not using transient arrays at all for this.
The return code is always "6F00 - no precise diagnosis" which leads to an unreferred byte[] or something like that, which I cannot find any except of the DF objects, that get instanciated when the new Object is created.
EDIT: just figured out it might be a more "general" problem and that is what I am doing wrong.
Now, if I take a "Hello World" like http://umer555.wordpress.com/2012/05/17/java-card-hello-world-applet/ and add some INS like I did here:
public class HalloWeltApplet extends Applet {
private static byte[] helloWorld = new byte[11];
private static final byte HW_CLA = (byte)0x80;
private static final byte HW_INS = (byte)0x00;
private static final byte HW_INS1 = (byte)0x01;
private static final byte HW_INS2 = (byte)0x02;
public static void install(byte[] bArray, short bOffset, byte bLength) {
new HalloWeltApplet().register(bArray, (short) (bOffset + 1), bArray[bOffset]);
}
public void process(APDU apdu) {
if (selectingApplet()) {
return;
}
byte[] buffer = apdu.getBuffer();
byte CLA = (byte) (buffer[ISO7816.OFFSET_CLA] & 0xFF);
byte INS = (byte) (buffer[ISO7816.OFFSET_INS] & 0xFF);
if(CLA != HW_CLA) {
ISOException.throwIt(ISO7816.SW_CLA_NOT_SUPPORTED);
}
switch(INS) {
case HW_INS:
getHelloWorld(apdu);
break;
case HW_INS1:
getHelloWorld1(apdu);
break;
case HW_INS2:
getHelloWorld2(apdu);
break;
default:
ISOException.throwIt(ISO7816.SW_INS_NOT_SUPPORTED);
}
}
private void getHelloWorld( APDU apdu) {
byte[] buffer = apdu.getBuffer();
short length = (short) helloWorld.length;
byte[] test = {(byte)'H',(byte)'e',(byte)'l',(byte)'l',(byte)'o',(byte)' ',(byte)'W',(byte)'o',(byte)'r',(byte)'l',(byte)'d',};
Util.arrayCopy(test, (short) 0, helloWorld, (short) 0, (short) test.length);
}
private void getHelloWorld1( APDU apdu) {
byte[] buffer = apdu.getBuffer();
short length = (short) helloWorld.length;
byte[] test = {(byte)'H',(byte)'i',(byte)' ',(byte)'W',(byte)'o',(byte)'r',(byte)'l',(byte)'d'};
Util.arrayCopy(test, (short) 0, helloWorld, (short) 0, (short) test.length);
}
private void getHelloWorld2( APDU apdu) {
byte[] buffer = apdu.getBuffer();
apdu.setOutgoing();
apdu.setOutgoingLength((short) helloWorld.length);
apdu.sendBytesLong(helloWorld, (short) 0, (short) helloWorld.length);
}
}
So this should in my eyes save 'Hello World' or 'Hi World' into helloWorld and with INS2 I can show which one is saved. But whenever I restart the program, helloWorld will be empty due to initiation, right? Could that be my problem after all, and if so, how can it be resolved?
The problem is probably here:
super(aid);
this.aid = aid;
First you correctly copy the data, then you overwrite the field in File with the one you've used in the DirectoryFile constructor. If that is a transient buffer or worse, the JCRE owned APDU buffer, then your code will fail as JCRE owned objects should not be used through persistent references.
Note that AID's are Application Identifiers. They identify applications such as your Java Card applet. Normally files and non-application DF's (especially child DF's) are not identified or selected with an AID but with a file identifier or a (related) short file identifier. See ISO/IEC 7816-4 (any version I guess) for details.
Note that resets work in the JCOP simulator, but that all information is lost when you restart the process; data is not saved to disk and applets must be reloaded.
Related
I have an Encrypted object in Minio, encrypted using the AES 128 bit CBC algorithm.
The object is quite large (~50 MB) so instead of loading it into the memory completely (which may cause out of memory exception), I am retrieving it in chunks of 1MB. I need to decrypt it before use.
Is it possible to decrypt the object in this way (1MB at a time, the whole object was encrypted in one go)?
If yes, how can I do it?
I have tried decrypting 16-byte chunks which produce the following errors:
javax.crypto.BadPaddingException: Given final block not properly padded
javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
To avoid an "out of memory error" you want to decrypt a large (encrypted) file in chunks of 1 mb size - yes it's possible with AES CBC mode.
Below you find a complete example that is generating a sample plaintext file ('plaintext.dat') with random content with the size of 50 mb + 1 byte (the + 1 byte is good to test for file sizes that are not exact multiples of 16 = AES blocksize).
In the next step this file is getting encrypted to 'ciphertext.dat' using a randomly created initialization vector and key.
The last step is the requested decryption method - it decrypts the encrypted file in chunks of 1 mb and in the lines '// obuf holds the decrypted chunk, do what you want to do with the data' and '// final data' you do have the decrypted data in the byte array obuf. For testing I'm writing the decrypted data to the file 'decryptedtext.dat' in appending mode (for that reason this file is deleted in the beginning if it exists).
To prove that decryption was successful I'm comparing the SHA256-hashes of plaintext- and decryptedtext-files.
Two notes: I'm using a 32 byte = 256 bit long key for AES CBC 256. This program has no proper exception handling and is for educational purposes only.
result:
decrypt AES CBC 256 in 1 mb chunks
file with random data created: plaintext.dat
encryption to ciphertext.dat was successfull: true
decryption in chunks of 1 mb
decrypted file written to decryptedtext.dat
plaintext equals decrytedtext file: true
code:
import javax.crypto.*;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.*;
import java.nio.file.Files;
import java.security.*;
import java.util.Arrays;
public class AES_CBC_chunk_decryption {
public static void main(String[] args) throws IOException, InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException,
InvalidAlgorithmParameterException, BadPaddingException, IllegalBlockSizeException {
System.out.println("https://stackoverflow.com/questions/63325528/decrypt-in-chunks-a-aes-128-cbc-encrypted-object/63325529#63325529");
System.out.println("decrypt AES CBC 256 in 1 mb chunks");
// setup for creation of a 50mb encrypted file
int filesize = (50 * 1024 * 1024) + 1; // 50 mb + 1 byte = 52428801 bytes
String filenamePlaintext = "plaintext.dat";
String filenameCiphertext = "ciphertext.dat";
String filenameDecryptedtext = "decryptedtext.dat";
File file = new File("plaintext.dat");
// fill with random bytes.
try (FileOutputStream out = new FileOutputStream(file)) {
byte[] bytes = new byte[filesize];
new SecureRandom().nextBytes(bytes);
out.write(bytes);
}
System.out.println("\nfile with random data created: " + filenamePlaintext);
// delete decrypted file if it exists
Files.deleteIfExists(new File(filenameDecryptedtext).toPath());
// setup random key & iv
SecureRandom secureRandom = new SecureRandom();
byte[] iv = new byte[16];
byte[] key = new byte[32]; // I'm using a 32 byte = 256 bit long key for aes 256
secureRandom.nextBytes(iv);
secureRandom.nextBytes(key);
// encrypt complete file
boolean resultEncryption = encryptCbcFileBufferedCipherOutputStream(filenamePlaintext, filenameCiphertext, key, iv);
System.out.println("encryption to " + filenameCiphertext + " was successfull: " + resultEncryption);
// encrypted file is 52428816 bytes long
System.out.println("\ndecryption in chunks of 1 mb");
// decryption in chunks of 1 mb
try (FileInputStream in = new FileInputStream(filenameCiphertext)) {
byte[] ibuf = new byte[(1024 * 1024)]; // chunks of 1 mb
int len;
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
SecretKeySpec secretKeySpec = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpec = new IvParameterSpec(iv);
cipher.init(Cipher.DECRYPT_MODE, secretKeySpec, ivParameterSpec);
while ((len = in.read(ibuf)) != -1) {
byte[] obuf = cipher.update(ibuf, 0, len);
if (obuf != null)
// obuf holds the decrypted chunk, do what you want to do with the data
// I'm writing it to a file in appending mode
try (FileOutputStream output = new FileOutputStream(filenameDecryptedtext, true)) {
output.write(obuf);
}
}
byte[] obuf = cipher.doFinal();
if (obuf != null)
// final data
try (FileOutputStream output = new FileOutputStream(filenameDecryptedtext, true)) {
output.write(obuf);
}
}
System.out.println("decrypted file written to " + filenameDecryptedtext);
System.out.println("plaintext equals decrytedtext file: " + filecompareSha256Large(filenamePlaintext, filenameDecryptedtext));
}
public static boolean encryptCbcFileBufferedCipherOutputStream(String inputFilename, String outputFilename, byte[] key, byte[] iv)
throws IOException, NoSuchPaddingException, NoSuchAlgorithmException, InvalidKeyException, InvalidAlgorithmParameterException {
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
try (FileInputStream in = new FileInputStream(inputFilename);
FileOutputStream out = new FileOutputStream(outputFilename);
CipherOutputStream encryptedOutputStream = new CipherOutputStream(out, cipher);) {
SecretKeySpec secretKeySpec = new SecretKeySpec(key, "AES");
IvParameterSpec ivParameterSpec = new IvParameterSpec(iv);
cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec, ivParameterSpec);
byte[] buffer = new byte[8096];
int nread;
while ((nread = in.read(buffer)) > 0) {
encryptedOutputStream.write(buffer, 0, nread);
}
encryptedOutputStream.flush();
}
if (new File(outputFilename).exists()) {
return true;
} else {
return false;
}
}
public static boolean filecompareSha256Large(String filename1, String filename2) throws IOException, NoSuchAlgorithmException {
boolean result = false;
byte[] hash1 = generateSha256Buffered(filename1);
byte[] hash2 = generateSha256Buffered(filename2);
result = Arrays.equals(hash1, hash2);
return result;
}
private static byte[] generateSha256Buffered(String filenameString) throws IOException, NoSuchAlgorithmException {
// even for large files
byte[] buffer = new byte[8192];
int count;
MessageDigest md = MessageDigest.getInstance("SHA-256");
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filenameString));
while ((count = bis.read(buffer)) > 0) {
md.update(buffer, 0, count);
}
bis.close();
return md.digest();
}
}
Yes, with AES-128-CBC, it is possible to decrypt just a single block of cyphertext. Each block is 128 bits (16 bytes).
See the diagram at https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_block_chaining_(CBC). As you can see, to decrypt any block of ciphertext, you AES-decrypt the block of ciphertext, then XOR the plaintext with the previous block of ciphertext. (For the first block, the plaintext is XOR'd with the IV).
The library that you are using is probably throwing these exceptions, because it is checking if the decrypted ciphertext is properly padded. Of course, if you are decrypting just one arbitrary block of ciphertext, it will not have the proper padding. However, you can use a tool like openssl to decrypt a single block of ciphertext, given the ciphertext, the key, and the previous block of ciphertext, like so:
echo -n 'bc6d8afc78e805b7ed7551e42da4d877' | xxd -p -r | openssl aes-128-cbc -d -nopad -K e3e33d2d9591b462c55503f7ec697839 -iv 1d3fa2b7c9008e1cdbc76a1f22388b89
where:
bc6d8afc78e805b7ed7551e42da4d877 is the block of ciphertext that you want to decrypt
e3e33d2d9591b462c55503f7ec697839 is the key
1d3fa2b7c9008e1cdbc76a1f22388b89 is the previous block of ciphertext
Yes, it is possible. However, due to the mode and padding it may be trickier to program than it looks at first sight.
However, I've created a class that will happily decode from any offset and to any size. Note that the ciphertext should not contain the IV.
In hindsight I might better have used ByteBuffer to make it a bit more flexible, but yeah, that will require an entire rewrite...
package com.stackexchange.so;
import java.security.GeneralSecurityException;
import java.security.InvalidAlgorithmParameterException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.SecretKey;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
/**
* A class that helps you to partially decrypt a CBC ciphertext. Although this class helps you to partially decrypt any
* part, you'd probably want to decrypt chunks that consists of a specific number of blocks; both the <code>off</code>
* and <code>len</code> parameter should be a modulus the block size. If you know the exact plaintext length then you
* can size the last chunk precisely.
*
* #author maartenb
*/
public class CBCDecryptByOffset {
private enum State {
UNINITIALIZED, INITIALIZED, RUNNING;
};
private final Cipher cbcCipher;
private SecretKey symKey;
private IvParameterSpec iv;
private State state = State.UNINITIALIZED;
/**
* Creates the CBC decryptor class and initializes it.
* #param blockCipher the block cipher, without block cipher mode or padding indication e.g. <code>"AES"</code>
* #throws NoSuchAlgorithmException if the block cipher is not available for <code>"CBC"</code>
* #throws NoSuchPaddingException if the block cipher in CBC mode is not available with <code>"NoPadding"</code>
*/
public CBCDecryptByOffset(String blockCipher) throws NoSuchAlgorithmException, NoSuchPaddingException {
this.cbcCipher = Cipher.getInstance(blockCipher + "/CBC/NoPadding");
}
/**
* Mimics {#link Cipher#init(int, java.security.Key, java.security.spec.AlgorithmParameterSpec)} except that it
* doesn't include options for encryption, wrapping or unwrapping.
*
* #param symKey the key to use
* #param iv the IV to use
* #throws InvalidKeyException if the key is not valid for the block cipher
* #throws InvalidAlgorithmParameterException if the IV is not valid for CBC, i.e. is not the block size
*/
public void init(SecretKey symKey, IvParameterSpec iv)
throws InvalidKeyException, InvalidAlgorithmParameterException {
this.symKey = symKey;
this.iv = iv;
// init directly, probably we want to start here, and it will perform a cursory check of the key and IV
this.cbcCipher.init(Cipher.DECRYPT_MODE, symKey, iv);
this.state = State.INITIALIZED;
}
/**
* Decrypts a partial number of bytes from a CBC encrypted ciphertext with PKCS#7 compatible padding.
*
* #param fullCT the full ciphertext
* #param off the offset within the full ciphertext to start decrypting
* #param len the amount of bytes to decrypt
* #return the plaintext of the partial decryption
* #throws BadPaddingException if the ciphertext is not correctly padded (only checked for the final CT block)
* #throws IllegalBlockSizeException if the ciphertext is empty or not a multiple of the block size
*/
public byte[] decryptFromOffset(byte[] fullCT, int off, int len)
throws BadPaddingException, IllegalBlockSizeException {
if (state == State.UNINITIALIZED) {
throw new IllegalStateException("Instance should be initialized before decryption");
}
int n = cbcCipher.getBlockSize();
if (fullCT.length == 0 || fullCT.length % n != 0) {
throw new IllegalBlockSizeException(
"Ciphertext must be a multiple of the blocksize, and should contain at least one block");
}
if (off < 0 || off > fullCT.length) {
throw new IllegalArgumentException("Invalid offset: " + off);
}
if (len < 0 || off + len < 0 || off + len > fullCT.length) {
throw new IllegalArgumentException("Invalid len");
}
if (len == 0) {
return new byte[0];
}
final int blockToDecryptFirst = off / n;
final int blockToDecryptLast = (off + len - 1) / n;
final int bytesToDecrypt = (blockToDecryptLast - blockToDecryptFirst + 1) * n;
final byte[] pt;
try {
// determine the IV to use
if (state != State.INITIALIZED || off != 0) {
IvParameterSpec vector;
final int blockWithVector = blockToDecryptFirst - 1;
if (blockWithVector == -1) {
vector = iv;
} else {
vector = new IvParameterSpec(fullCT, blockWithVector * n, n);
}
cbcCipher.init(Cipher.DECRYPT_MODE, symKey, vector);
}
// perform the actual decryption (note that offset and length are in bytes)
pt = cbcCipher.doFinal(fullCT, blockToDecryptFirst * n, bytesToDecrypt);
} catch (GeneralSecurityException e) {
throw new RuntimeException("Incorrectly programmed, error should never appear", e);
}
// we need to unpad if the last block is the final ciphertext block
int sigPadValue = 0;
final int finalCiphertextBlock = (fullCT.length - 1) / n;
if (blockToDecryptLast == finalCiphertextBlock) {
int curPaddingByte = bytesToDecrypt - 1;
int padValue = Byte.toUnsignedInt(pt[curPaddingByte]);
if (padValue == 0 || padValue > n) {
throw new BadPaddingException("Invalid padding");
}
for (int padOff = curPaddingByte - 1; padOff > curPaddingByte - padValue; padOff--) {
if (Byte.toUnsignedInt(pt[padOff]) != padValue) {
throw new BadPaddingException("Invalid padding");
}
}
// somebody tries to decrypt just padding bytes
if (off >= (blockToDecryptLast + 1) * n - padValue) {
sigPadValue = len;
} else {
// calculate if any (significant) padding bytes need to be ignored within the plaintext
int bytesInFinalBlock = (off + len - 1) % n + 1;
sigPadValue = padValue - (n - bytesInFinalBlock);
if (sigPadValue < 0) {
sigPadValue = 0;
}
}
}
int ptStart = off - blockToDecryptFirst * n;
int ptSize = len - sigPadValue;
state = State.RUNNING;
if (pt.length == ptSize) {
return pt;
}
return Arrays.copyOfRange(pt, ptStart, ptStart + ptSize);
}
}
Note that I've tested the general functionality but I'd make sure that I wrap it with some JUnit tests if I were you.
I've encountered a problem while declaring a BigNumber datatype in my javacard applet. The applet loads properly into the simulator if i just comment the declaration. The problem to be precise is while loading the import.cap file
(jcshell: Error code: 6a80 (Wrong data))
java card kit 2.2.2 using
import javacard.framework.APDU;
import javacard.framework.Applet;
import javacard.framework.ISO7816;
import javacard.framework.ISOException;
import javacard.framework.JCSystem;
import javacardx.framework.math.BigNumber;
public class LargeBal extends Applet {
// CLA byte
public static final byte BANK_CLA = (byte) 0x80;
// INS byte
public static final byte INS_GET_BALANCE = 0X02;
public static final byte INS_CREDIT = 0X04;
public static final byte INS_DEBIT = 0X06;
/**
* SW bytes for Arithmetic exception
*/
final static short INVALID_NUMBER_FORMAT = 0x6308;
/**
* Initial account balance
*/
final static byte[] INITIAL_ACCOUNT_BALANCE = { (byte) 0x01, (byte) 0x00,
(byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00 };
// Amount of money in user's account
private BigNumber accountBalance;
// Big number for temporary calculation
BigNumber tempBigNum;
// temporary buffer used as scratch space
byte[] scratchSpace;
private LargeBal() {
accountBalance = new BigNumber((byte) 8);
// initialize account balance to 100,000.00
accountBalance.init(INITIAL_ACCOUNT_BALANCE, (byte) 0,
(byte) INITIAL_ACCOUNT_BALANCE.length, BigNumber.FORMAT_BCD);
// initialize the temporary big number
tempBigNum = new BigNumber(BigNumber.getMaxBytesSupported());
// initialize the scratchSpace
scratchSpace = JCSystem.makeTransientByteArray((short) 10,
JCSystem.CLEAR_ON_DESELECT);
register();
}
public static void install(byte[] bArray, short bOffset, byte bLength) {
// GP-compliant JavaCard applet registration
new LargeBal();
}
public void process(APDU apdu) {
// Good practice: Return 9000 on SELECT
if (selectingApplet()) {
return;
}
byte[] buf = apdu.getBuffer();
switch (buf[ISO7816.OFFSET_INS]) {
case INS_GET_BALANCE:
getBalance(apdu, buf);
break;
case INS_CREDIT:
break;
case INS_DEBIT:
break;
default:
// good practice: If you don't know the INStruction, say so:
ISOException.throwIt(ISO7816.SW_INS_NOT_SUPPORTED);
}
}
private void getBalance(APDU apdu, byte[] buffer) {
if (buffer[ISO7816.OFFSET_P1] == BigNumber.FORMAT_BCD) {
accountBalance.toBytes(buffer, (short) 0, (short) 8,
BigNumber.FORMAT_BCD);
} else if (buffer[ISO7816.OFFSET_P1] == BigNumber.FORMAT_HEX) {
accountBalance.toBytes(buffer, (short) 0, (short) 8,
BigNumber.FORMAT_HEX);
} else
ISOException.throwIt(INVALID_NUMBER_FORMAT);
apdu.setOutgoingAndSend((short) 0, (short) 8);
}
}
javacardx.framework.math is an optional package. Thus, not all cards/emulators implement this. In your case, it seems that the card does not implement javacardx.framework.math.BigNumber. Consequently it refuses to load/install the applet.
From the Runtime Environment Specification, Java Card Platform, Version 2.2.2 (section 9.7):
Optional Extension Packages
Some API packages in the Java Card technology are designated as extension
packages and may be optionally supported by an implementation. But, if supported,
all the classes in the package and its subpackages must be implemented by the
platform and reside on the card.
The following are optional Java Card technology extension packages:
javacardx.apdu [...]
javacardx.biometry [...]
javacardx.crypto [...]
javacardx.external [...]
javacardx.framework [...] If implemented, this package must include all the contained subpackages - util, math, and tlv.
I have a rest service written to receive a file and save it.
The problem is that when I receive more than 2 requests, the files are not written only the last request is taken into consideration and written.
Here is my code:
#POST
#RequestMapping(value = "/media/{mediaName}/{mediaType}")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
#ResponseBody
public String updateResourceLocally(#FormDataParam("rawData") InputStream rawData, #PathVariable("mediaName") String mediaName, #PathVariable("mediaType") String mediaType) {
logger.info("Entering updateResourceLocally for " + jobId + "; for media type: " + mediaType);
final String storeDir = "/tmp/test/" + mediaName + ("/");
final String finalExtension = mediaType;
final InputStream finalRawData = rawData;
// new Thread(new Runnable() {
// public void run() {
// writeToFile(finalRawData, storeDir, finalExtension);
// }
// }).start();
writeToFile(finalRawData, storeDir, finalExtension);
// int poolSize = 100;
// ExecutorService executor = Executors.newFixedThreadPool(poolSize);
// executor.execute(new Runnable() {
// #Override
// public void run() {
// writeToFile(rawData, storeDir, finalExtension);
// }
// });
logger.info("File uploaded to : " + storeDir);
return "Success 200";
}
I tried to put the writeToFile into threads, but still no success. Here is what writeToFile does
public synchronized void writeToFile(InputStream rawData,
String uploadedFileLocation, String extension) {
StringBuilder finalFileName = null;
String currentIncrement = "";
String fileName = "raw";
try {
File file = new File(uploadedFileLocation);
if (!file.exists()) {
file.mkdirs();
}
while (true) {
finalFileName = new StringBuilder(fileName);
if (!currentIncrement.equals("")) {
finalFileName.append("_").append(currentIncrement).append(extension);
}
File f = new File(uploadedFileLocation + finalFileName);
if (f.exists()) {
if (currentIncrement.equals("")) {
currentIncrement = "1";
} else {
currentIncrement = (Integer.parseInt(currentIncrement) + 1) + "";
}
} else {
break;
}
}
int read = 0;
byte[] bytes = new byte[1024];
OutputStream out = new FileOutputStream(new File(uploadedFileLocation + finalFileName));
while ((read = rawData.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
} catch (IOException e) {
throw new RuntimeException(e.getMessage());
}
}
The writeToFile creates a folder and writes a file, if the file already exists, it appends 1 and then increments the 1 accordingly and writes the file, so I would get raw.zip, raw-1.zip, etc.
I think the inputstream bytes are being lost, am I correct in my assumption?
NOTE: I do not have a UI client, I am using Poster a Firefox extension.
Update: What I am trying to achieve here is very simple
I receive number of requests with files attached
I need to save them. If the mediaName and mediaType are the same, then I need to append something to the filename and save it in the same location
If they are different I do not have a problem
The problem I am facing with the current code is that, when I post multiple time to the same URL, I have file-names created according to what I want, but the file content is not right, they vary depending on when the request came in and only the last POST's request is written properly.
Eg. I have a zip file of size 250MB, when I post 5 time, the 1st four will have random sizes and the 5th will have the complete 250MB, but the previous four should also have the same content.
You must separate the stream copy from the free filename assignation. The stream copy must be done within the calling thread (jersey service). Only the file naming operation must be common to all threads/requests.
Here is your code with a little refactoring :
getNextFilename
This file naming operation must be synchronized to guarantee each call gives a free name. This functions creates an empty file to guarantee the next call to work, because the function relies on file.exists().
public synchronized File getNextFilename(String uploadedFileLocation, String extension)
throws IOException
{
// This function MUST be synchronized to guarantee unicity of files names
// Synchronized functions must be the shortest possible to avoid threads waiting each other.
// No long job such as copying streams here !
String fileName = "raw";
//Create directories (if not already existing)
File dir = new File(uploadedFileLocation);
if (!dir.exists())
dir.mkdirs();
//Search for next free filename (raw.<extension>, else raw_<increment>.<extension>)
int currentIncrement = 0;
String finalFileName = fileName + "." + extension;
File f = new File(uploadedFileLocation + finalFileName);
while (f.exists())
{
currentIncrement++;
finalFileName = fileName + "_" + currentIncrement + "." + extension;
f = new File(uploadedFileLocation + finalFileName);
}
//Creates the file with size 0 in order to physically reserve the file "raw_<n>.extension",
//so the next call to getNextFilename will find it (f.exists) and will return "raw_<n+1>.extension"
f.createNewFile();
//The file exists, let the caller fill it...
return f;
}
writeToFile
Must not be synchronized !
public void writeToFile(InputStream rawData, String uploadedFileLocation, String extension)
throws IOException
{
//(1) Gets next available filename (creates the file with 0 size)
File file = getNextFilename(uploadedFileLocation, extension);
//(2) Copies data from inputStream to file
int read = 0;
byte[] bytes = new byte[1024];
OutputStream out = new FileOutputStream(file);
while ((read = rawData.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
}
How I can convert the following methods to C# 2.0?
private static string ToHexString(byte[] bytes)
{
return string.Join(string.Empty, bytes.Select(x => x.ToString("X2")).ToArray());
}
private static byte[] ToByteArray(string hex)
{
return Enumerable.Range(0, hex.Length).
Where(x => 0 == x % 2).
Select(x => Convert.ToByte(hex.Substring(x, 2), 16)).
ToArray();
}
I haven't got experience with .NET 2.0. Thanks!
void Main()
{
string s = ToHexString(new byte[]{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15});
Console.WriteLine(s);
foreach (byte b in ToByteArray(s))
Console.WriteLine(b);
}
private static string ToHexString(byte[] bytes)
{
StringBuilder sb = new StringBuilder();
foreach (byte b in bytes)
sb.Append(b.ToString("X2"));
return sb.ToString();
}
private static byte[] ToByteArray(string hex)
{
byte[] b = new byte[hex.Length/2];
for (int i = 0; i < b.Length; i++)
{
b[i] = Convert.ToByte(hex.Substring(i*2,2), 16);
}
return b;
}
You should be able to do this conversion yourself. Obviously you'll want to convert it into a for loop. Enumerable.Range essentially provides an int[] array upon which to loop. After that, the Where equates to an if check, the Select is a transformation from the int to a substring and finally to a byte, and last, you stuff all that into an array, probably by adding them to a List<byte> declared outside the loop. (and when you're done, you can call ToArray on the list)
I could provide a complete answer, but I think this sort of exercise is best left to you so you can learn from it.
Background
Our Eclipse RCP 3.6-based application lets people drag files in for storage/processing. This works fine when the files are dragged from a filesystem, but not when people drag items (messages or attachments) directly from Outlook.
This appears to be because Outlook wants to feed our application the files via a FileGroupDescriptorW and FileContents, but SWT only includes a FileTransfer type. (In a FileTransfer, only the file paths are passed, with the assumption that the receiver can locate and read them. The FileGroupDescriptorW/FileContents approach can supply files directly application-to-application without writing temporary files out to disk.)
We have tried to produce a ByteArrayTransfer subclass that could accept FileGroupDescriptorW and FileContents. Based on some examples on the Web, we were able to receive and parse the FileGroupDescriptorW, which (as the name implies) describes the files available for transfer. (See code sketch below.) But we have been unable to accept the FileContents.
This seems to be because Outlook offers the FileContents data only as TYMED_ISTREAM or TYMED_ISTORAGE, but SWT only understands how to exchange data as TYMED_HGLOBAL. Of those, it appears that TYMED_ISTORAGE would be preferable, since it's not clear how TYMED_ISTREAM could provide access to multiple files' contents.
(We also have some concerns about SWT's desire to pick and convert only a single TransferData type, given that we need to process two, but we think we could probably hack around that in Java somehow: it seems that all the TransferDatas are available at other points of the process.)
Questions
Are we on the right track here? Has anyone managed to accept FileContents in SWT yet? Is there any chance that we could process the TYMED_ISTORAGE data without leaving Java (even if by creating a fragment-based patch to, or a derived version of, SWT), or would we have to build some new native support code too?
Relevant code snippets
Sketch code that extracts file names:
// THIS IS NOT PRODUCTION-QUALITY CODE - FOR ILLUSTRATION ONLY
final Transfer transfer = new ByteArrayTransfer() {
private final String[] typeNames = new String[] { "FileGroupDescriptorW", "FileContents" };
private final int[] typeIds = new int[] { registerType(typeNames[0]), registerType(typeNames[1]) };
#Override
protected String[] getTypeNames() {
return typeNames;
}
#Override
protected int[] getTypeIds() {
return typeIds;
}
#Override
protected Object nativeToJava(TransferData transferData) {
if (!isSupportedType(transferData))
return null;
final byte[] buffer = (byte[]) super.nativeToJava(transferData);
if (buffer == null)
return null;
try {
final DataInputStream in = new DataInputStream(new ByteArrayInputStream(buffer));
long count = 0;
for (int i = 0; i < 4; i++) {
count += in.readUnsignedByte() << i;
}
for (int i = 0; i < count; i++) {
final byte[] filenameBytes = new byte[260 * 2];
in.skipBytes(72); // probable architecture assumption(s) - may be wrong outside standard 32-bit Win XP
in.read(filenameBytes);
final String fileNameIncludingTrailingNulls = new String(filenameBytes, "UTF-16LE");
int stringLength = fileNameIncludingTrailingNulls.indexOf('\0');
if (stringLength == -1)
stringLength = 260;
final String fileName = fileNameIncludingTrailingNulls.substring(0, stringLength);
System.out.println("File " + i + ": " + fileName);
}
in.close();
return buffer;
}
catch (final Exception e) {
return null;
}
}
};
In the debugger, we see that ByteArrayTransfer's isSupportedType() ultimately returns false for the FileContents because the following test is not passed (since its tymed is TYMED_ISTREAM | TYMED_ISTORAGE):
if (format.cfFormat == types[i] &&
(format.dwAspect & COM.DVASPECT_CONTENT) == COM.DVASPECT_CONTENT &&
(format.tymed & COM.TYMED_HGLOBAL) == COM.TYMED_HGLOBAL )
return true;
This excerpt from org.eclipse.swt.internal.ole.win32.COM leaves us feeling less hope for an easy solution:
public static final int TYMED_HGLOBAL = 1;
//public static final int TYMED_ISTORAGE = 8;
//public static final int TYMED_ISTREAM = 4;
Thanks.
even if
//public static final int TYMED_ISTREAM = 4;
Try below code.. it should work
package com.nagarro.jsag.poc.swtdrag;
imports ...
public class MyTransfer extends ByteArrayTransfer {
private static int BYTES_COUNT = 592;
private static int SKIP_BYTES = 72;
private final String[] typeNames = new String[] { "FileGroupDescriptorW", "FileContents" };
private final int[] typeIds = new int[] { registerType(typeNames[0]), registerType(typeNames[1]) };
#Override
protected String[] getTypeNames() {
return typeNames;
}
#Override
protected int[] getTypeIds() {
return typeIds;
}
#Override
protected Object nativeToJava(TransferData transferData) {
String[] result = null;
if (!isSupportedType(transferData) || transferData.pIDataObject == 0)
return null;
IDataObject data = new IDataObject(transferData.pIDataObject);
data.AddRef();
// Check for descriptor format type
try {
FORMATETC formatetcFD = transferData.formatetc;
STGMEDIUM stgmediumFD = new STGMEDIUM();
stgmediumFD.tymed = COM.TYMED_HGLOBAL;
transferData.result = data.GetData(formatetcFD, stgmediumFD);
if (transferData.result == COM.S_OK) {
// Check for contents format type
long hMem = stgmediumFD.unionField;
long fileDiscriptorPtr = OS.GlobalLock(hMem);
int[] fileCount = new int[1];
try {
OS.MoveMemory(fileCount, fileDiscriptorPtr, 4);
fileDiscriptorPtr += 4;
result = new String[fileCount[0]];
for (int i = 0; i < fileCount[0]; i++) {
String fileName = handleFile(fileDiscriptorPtr, data);
System.out.println("FileName : = " + fileName);
result[i] = fileName;
fileDiscriptorPtr += BYTES_COUNT;
}
} catch (Exception e) {
e.printStackTrace();
} finally {
OS.GlobalFree(hMem);
}
}
} finally {
data.Release();
}
return result;
}
private String handleFile(long fileDiscriptorPtr, IDataObject data) throws Exception {
// GetFileName
char[] fileNameChars = new char[OS.MAX_PATH];
byte[] fileNameBytes = new byte[OS.MAX_PATH];
COM.MoveMemory(fileNameBytes, fileDiscriptorPtr, BYTES_COUNT);
// Skip some bytes.
fileNameBytes = Arrays.copyOfRange(fileNameBytes, SKIP_BYTES, fileNameBytes.length);
String fileNameIncludingTrailingNulls = new String(fileNameBytes, "UTF-16LE");
fileNameChars = fileNameIncludingTrailingNulls.toCharArray();
StringBuilder builder = new StringBuilder(OS.MAX_PATH);
for (int i = 0; fileNameChars[i] != 0 && i < fileNameChars.length; i++) {
builder.append(fileNameChars[i]);
}
String name = builder.toString();
try {
File file = saveFileContent(name, data);
if (file != null) {
System.out.println("File Saved # " + file.getAbsolutePath());
;
}
} catch (IOException e) {
System.out.println("Count not save file content");
;
}
return name;
}
private File saveFileContent(String fileName, IDataObject data) throws IOException {
File file = null;
FORMATETC formatetc = new FORMATETC();
formatetc.cfFormat = typeIds[1];
formatetc.dwAspect = COM.DVASPECT_CONTENT;
formatetc.lindex = 0;
formatetc.tymed = 4; // content.
STGMEDIUM stgmedium = new STGMEDIUM();
stgmedium.tymed = 4;
if (data.GetData(formatetc, stgmedium) == COM.S_OK) {
file = new File(fileName);
IStream iStream = new IStream(stgmedium.unionField);
iStream.AddRef();
try (FileOutputStream outputStream = new FileOutputStream(file)) {
int increment = 1024 * 4;
long pv = COM.CoTaskMemAlloc(increment);
int[] pcbWritten = new int[1];
while (iStream.Read(pv, increment, pcbWritten) == COM.S_OK && pcbWritten[0] > 0) {
byte[] buffer = new byte[pcbWritten[0]];
OS.MoveMemory(buffer, pv, pcbWritten[0]);
outputStream.write(buffer);
}
COM.CoTaskMemFree(pv);
} finally {
iStream.Release();
}
return file;
} else {
return null;
}
}
}
Have you looked at https://bugs.eclipse.org/bugs/show_bug.cgi?id=132514 ?
Attached to this bugzilla entry is an patch (against an rather old version of SWT) that might be of interest.
I had the same problem and created a small library providing a Drag'n Drop Transfer Class for JAVA SWT. It can be found here:
https://github.com/HendrikHoetker/OutlookItemTransfer
Currently it supports dropping Mail Items from Outlook to your Java SWT application and will provide a list of OutlookItems with the Filename and a byte array of the file contents.
All is pure Java and in-memory (no temp files).
Usage in your SWT java application:
if (OutlookItemTransfer.getInstance().isSupportedType(event.currentDataType)) {
Object o = OutlookItemTransfer.getInstance().nativeToJava(event.currentDataType);
if (o != null && o instanceof OutlookMessage[]) {
OutlookMessage[] outlookMessages = (OutlookMessage[])o;
for (OutlookMessage msg: outlookMessages) {
//...
}
}
}
The OutlookItem will then provide two elements: filename as String and file contents as array of byte.
From here on, one could write it to a file or further process the byte array.
To your question above:
- What you find in the file descriptor is the filename of the outlook item and a pointer to an IDataObject
- the IDataObject can be parsed and will provide an IStorage object
- The IStorageObject will be then a root container providing further sub-IStorageObjects or IStreams similar to a filesystem (directory = IStorage, file = IStream
You find those elements in the following lines of code:
Get File Contents, see OutlookItemTransfer.java, method nativeToJava:
FORMATETC format = new FORMATETC();
format.cfFormat = getTypeIds()[1];
format.dwAspect = COM.DVASPECT_CONTENT;
format.lindex = <fileIndex>;
format.ptd = 0;
format.tymed = TYMED_ISTORAGE | TYMED_ISTREAM | COM.TYMED_HGLOBAL;
STGMEDIUM medium = new STGMEDIUM();
if (data.GetData(format, medium) == COM.S_OK) {
// medium.tymed will now contain TYMED_ISTORAGE
// in medium.unionfield you will find the root IStorage
}
Read the root IStorage, see CompoundStorage, method readOutlookStorage:
// open IStorage object
IStorage storage = new IStorage(pIStorage);
storage.AddRef();
// walk through the content of the IStorage object
long[] pEnumStorage = new long[1];
if (storage.EnumElements(0, 0, 0, pEnumStorage) == COM.S_OK) {
// get storage iterator
IEnumSTATSTG enumStorage = new IEnumSTATSTG(pEnumStorage[0]);
enumStorage.AddRef();
enumStorage.Reset();
// prepare statstg structure which tells about the object found by the iterator
long pSTATSTG = OS.GlobalAlloc(OS.GMEM_FIXED | OS.GMEM_ZEROINIT, STATSTG.sizeof);
int[] fetched = new int[1];
while (enumStorage.Next(1, pSTATSTG, fetched) == COM.S_OK && fetched[0] == 1) {
// get the description of the the object found
STATSTG statstg = new STATSTG();
COM.MoveMemory(statstg, pSTATSTG, STATSTG.sizeof);
// get the name of the object found
String name = readPWCSName(statstg);
// depending on type of object
switch (statstg.type) {
case COM.STGTY_STREAM: { // load an IStream (=File)
long[] pIStream = new long[1];
// get the pointer to the IStream
if (storage.OpenStream(name, 0, COM.STGM_DIRECT | COM.STGM_READ | COM.STGM_SHARE_EXCLUSIVE, 0, pIStream) == COM.S_OK) {
// load the IStream
}
}
case COM.STGTY_STORAGE: { // load an IStorage (=SubDirectory) - requires recursion to traverse the sub dies
}
}
}
}
// close the iterator
enumStorage.Release();
}
// close the IStorage object
storage.Release();